Craig has been blending UX, Analytics, AB Testing and Customer Insight methods for over 12 years, optimising growth for companies like Google, Amazon, Spotify, Eurostar, Lego, Asos, John Lewis and Red Bull.
He has trained hundreds of people to use ridiculously simple optimisation techniques, worked with over 400 clients, and helped them all to unlock latent value and growth inside their products or businesses. Using a mixture of Design Thinking, Lean, Agile, analytics, user research, A/B testing and cultural change, he’s driven over £4bn in incremental revenue for his clients.
1 / 143 Use ← → to navigate
Auto-generated transcript - may contain errors.
Tap a timestamp to jump the video.
Good afternoon, Edinburgh. It's a pleasure to be here. Little known fact little known fact here, I used to live in Edinburgh, down in Leith, when pubs had sawdust on the floor, not as some hipster affectation of days gone by where guys had lumberjack shirts, but because it was there to mop up all the blood, the beer and the teeth at the end of the evening and into one nice wee pile.
And definitely, Leith does not have the same frisson of danger as it used to have in the late eighties, when my main mission in life was to go out drinking and avoid getting my heat kicked in by someone like Begbie. So I'm glad to be back.
And what am I talking about today? Well, I'm writing a whole series of articles all about the death of things. Some of these Jeff already touched on. Some are even more disastrous in ways that involve human death, but it's all about death. And I'm picking one topic today, which is death by design of defective device experiences.
And I say to people at the start of this talk, this will be the best fifteen minutes, the most rewarding and profitable fifteen minutes you spend in Excel this year. And quite rightly, because all you tech people are really cynical ********, you don't believe me.
But I hope to prove it to you by the end of this talk. And there's a little teaser here for those people who are maybe thinking this isn't true, and I'll come to this at the end. Can you guess what these figures actually refer to?
And we'll come to that. But killing your growth comes with an explanation. What the hell is wrong here? Why don't people complain about this ****? How to mine your own website data to figure out what people have? How to get with customer centric testing?
Fix costly conversion killers quickly, and then automate all this stuff, because you'll break it again. And all it boils down to, the whole presentation is making your ****, whatever you make, work on their ****. Sounds easy. It's not. It's very hard. But ultimately, you're in charge of a choke point.
The device experience is the determinant of whether people can get in the ******* door of your shop. Right? If you're marketed to them and they can't get inside and experience the joys of your shop, then they're stuck outside, and your marketing is wasted.
The problem here with this kind of store is you think, well, that door looks okay, but would that door work if you were selling small children's clothing? Probably not, because I won't get a double buggy through there, and I probably wouldn't get a wheelchair through there as well.
So I have discriminated against groups of people, whether intentional or unintentional. This is my choke point, and I'm not letting them in through neglect or through laziness. And that's what today is about. So what the **** is wrong? Let me introduce you to the fungus that grows on my life and the stuff that I have to look at.
Everyone thinks my work is glamorous. No, it isn't. It involves wading through ***** like this every day. So I get I go to this Swedish airline site, and they say, like, nice little pop up. Would you like us to pre fill your passport information, mister Sullivan?
I go, yes, please. I would like that. And it fills it in for me, but it puts ******* stars in the passport number. And then it rejects it, says, the passport number's invalid. And I'm like, but you did this, you ********. It's not my fault.
Why are you giving me an error? I didn't actually do it. It's terrible stuff. British Airways IT seems to be in the press a lot. I've got several examples from them. Wonder why? Here's a hyperlink. Right? The simplest thing to **** up on the Internet. Right?
But British Airways have managed to break one of the underlying technologies of the Internet because the hyperlink doesn't work to fill in the booking reference in the last passenger name. But it gets worse. I was sitting on a freezing train station, and I saw this advert for Funcal caves in Madeira. It looked brilliant, visceral advertising, wicked.
Take me there to the b a dot com slash gatwick page, and what do I find? A terminal move at London Gatwick. Where is my holiday? So I scroll and scroll and scroll three quarters of way down the pages, some vaguely like it, But then I click on it, and I think, holy ****, what is this?
Right. Bear in mind, this is a mobile phone screen. You're seeing it somewhat magnified here, and you still can't ******* read it. What chance do I have when it's four inches high? Right? Seriously. But let's think about what does this look like in real life.
The problem is that the touch targets are one point four to two point five millimeters. I measured them. Right? And that's too small to have a reasonable error rate for people tapping them. Why is that? Because if you take it and superimpose what a finger looks like, let's take my giant ******* meat puppet and try and click something on there.
How's that working? Which one do you think I've clicked? Oh, what about those nice, close together pagination controls? Why don't I try and slap the meat puppet on those? That's going to work. And it gets worse. It gets even more annoying, because when I leave the website after not being able to book my ******* holiday, they say, read something, your feedback.
And I imagined that moment, we'd like to take your feedback up the behookie, because that's how I felt about it. But of course, I can't give them my feedback, because the yes, please button is off ******* canvas, and I can't click on it.
So the only option is no thanks, and that's what I thought about British Airways. Here's another bit of pain and ******* misery that you're putting into people's lives for absolutely no reason whatsoever. These are the password rules from an Oracle system. Now I remember why it's so horrible to use Oracle software.
But there's an important point here. Here's a new one from me, from KLM, right, where when you type in your password that you've had to construct for all these websites that need a really complicated password, and it's fifteen characters long, you come to this ******* site, and you see, this field can contain up to twelve characters.
And you're like, guys, guys, this is like twenty nineteen. Right? If you're trying to save disk space on your SQL database, you're in the wrong decade. So that that that really got me for a minute. But here's an interesting thing. The strength of your password does not come from its complexity.
It comes from its ******* length. So a nine character, easy to remember password, your mommy, right, or two words concatenated together, is three times more secure than the eight character, really complicated, I'm going to forget it next week and reset my password version.
And the thing is, attackers don't break into individual accounts. They break into the platform or exploit weaknesses in your web app security to get tons of these. I'm not interested in one. So it's a fallacy. You've shoved this cactus in your customers' eyes nominally to protect them for security, but it is not actually protecting them, in terms of security.
Here's another favorite one. I've had a running thing with NatWest Bank for for a long time, but for two and a half months, they had a bug in Chrome. You couldn't fix it anyway. Have you any idea how evil it is to say to you, you're gonna have to use ******* Internet Explorer to do your online banking for two months?
That's just evil. So because they're not supporting a standards based browser, they force me to use a non standards based browser? That is ridiculous. Come on, Nat West. Another common one, autofill. Autofill. Always going wrong, particularly with phone numbers and other pieces of data, but this one totally got me.
Eight AM in the morning, I'm booking a flight, I read the error message, invalid phone number format. Only digits are allowed. Only digits are allowed. Only digits are allowed. It's got digits in it. Is it too many? I've tried deleting some of them.
Is it too many digits? No. No. No. I tried changing the number a bit. No. No. It's the ******* zero. Right? It's the ******* zero. But wait a minute. Zero is a digit. I'm thinking, oh, this is getting really philosophical on me, eight AM in the morning.
This is not exactly what I anticipated, having a challenged response with the error handling inside a form. But that was it. It was the zero. You think, god, you know, programming must be really hard these days. You can't strip a leading zero off a string before validating it.
Must be really tough work. Here's another good one. This was a big one. We noticed on July, eleven months before I looked at the data, that all the page views increased in the product listings on this site. This site takes eight to ten million pounds a month.
Right? And what they've done is the page views increased because every single iPhone visiting the site, these filter and sort controls were broken. So all iPhone users, when they were going to get a new phone, were actually having to go through every page of all the phones.
Right? That cost them a lot of money. It sucks. And here's another one I hate. It's called PEE. Right? You know when you go to a field and it's got wee bits, and you start putting the stuff in, and it says, please enter or validate.
You're like, ******* hang on. I haven't even got to the year yet. Here's another one. My local parking application. Right? Here's this security code. You must enter a valid security code. I've only just gotten here. Give give me time. It feels like someone's pushing you with a cattle prod, and what this is is premature error ejaculation.
You've shot the error load too early. Please, please, get the timing right. So all these millions and millions of hours of friction, wasted time, stress hormones, missed appointments, order errors, things that don't arrive, all of that stuff just mounts up. Right? What about about machine learning?
Machine learning will save us from all of these problems. Well, I just pulled out a couple of random examples the last couple of weeks to show you how sophisticated it's become. All the money that LinkedIn put into their machine learning algorithms to surface content across your network and people that you might want to know in order to make it an interesting news feed.
And this is what I get. Congratulate Craig Sullivan for twelve years at Craig Sullivan Plastering Services. Why? The only thing I've got in common with are they suggesting a career change? Craig, your work is ****. Go and do some plastering. If that's the pinnacle of their recommendations engine, then it's dogshy.
Here's another one. Cora, digital assistant. You can bet that these guys spent at least three and a half million pounds implementing this, and if you do any chat on RBS systems, it's fronted up by this piece of crap. So I said, I try something simple.
My card isn't listed on my account. I can't use the Challenger Response Reader. And it says, oh, okay. I'm not sure. Could you reword your question? I thought that was quite clear. So I thought, maybe it's missing some important piece of information. So I'll read it and say, my card isn't listed on my business account.
That'll get you. And then it says, oh, I could try an FAQ search. Here's the most popular answer. How do I open an account? I've already ******* got one. I just told you, and it costs three and a half million. So the only thing that this understands that I type into it now is I want to speak to a human.
Wasting time. And even something like collaborative filtering, which in various guises has been around for fifteen years, is still not that great. It makes a lot of people money, but it's just not that brilliant. And in my opinion, most machine learning is just dumb and brutal, statistical modeling, and it's able to accelerate all of your inherent biases, magnify them, and massively scale up human stupidity at the same time.
But here's another example of something stupid, a new one for your list, a UX bug. This is when the developers or your external company you're using for development work tells you that this stuff is working, that it's code correct, that they have met their contractual requirements.
But when you start using it, you find that, oh, on certain phones, this button, this advert goes over the button, or you can't dismiss the pop ups, or you can't scroll the page, or the viewport's too small, or the touch targets are too close together.
It's not a bug, but people still can't bloody use it. And there are some very pernicious examples of these that hit almost every site. So someone asked me, Craig, are our customers really ******* stupid because their name seems to be the most edited field on our website, and we can't understand it.
And I said, well, usually, people, when they get to sort of four or five, they've got their name by then. Right? And they, you know, they don't tend to start getting it wrong when they're twenty. Right? But the problem was you left the ******* auto correct on.
Right? So even if you've got a vaguely non white Anglo Saxon Protestant name, well, that's you totally ******. Right? Because it's definitely gonna try and correct that to something else. So it's kind of like the slightly racist name form filling in a way, completely unintentional, still very annoying.
And eighty percent of the sites that Baymark tested had this problem of not setting the autocorrect attributes. Right? Fifty four percent of websites that they tested had the wrong keyboard type. If you're filling in a phone number, you don't want an alphanumeric keyboard.
You're not typing phonetically alpha zero nine or two. Right? You want numbers. Where's the numbers? And do you think that a five hundred and twenty one percent increase in the key hit area might increase conversion and reduce error rate in your forms? Of course it will.
It makes money. And you don't even have to do this. Beemard have given you a cheat sheet for every type of field that you can use on a form, and they tell you what attributes to set. And so all this stuff makes me really ******* angry.
It brings out the inner viking and flames in me. This is my friend Michael. He gets angry with that too, because it's simple. It's not hard to fix. This is my hierarchy of needs. It works on their devices. It loads and runs quickly.
They can actually read the text on the screen, and then they can use it. All the rest is marketing quiffle. Don't worry about these four bits, the other bit. That **** will take care of itself. That's what it is device compatibility, performance accessibility, usability, and some easy **** stacked on top called marketing.
But that easy **** won't work without these foundations, And if you're not doing these, all you're doing is you're burning rubber in the product car park, because there's lots of smoke, lots of light, lots of noise. Everyone has a brilliant time, but at the end of the evening, you're still in Tesco's car park, and you have no tires.
Right? That's called agility. It's a kind of drunken agility. But if it was that bad, customers would complain, wouldn't they? No. Have any of you complained about a rendering issue in Chrome to a site that you use? No. I doubt it. No one's got time.
So if five people complain, five people complain, is there a is there a massive problem? Yeah. It's probably gone ******* nuclear if five call you. But if only one person complains, does that mean there's there's a problem? What if nobody complains at all?
Does that mean you that everything's working fine? No. You can't rely on that. And the problem is is when I see stuff that I'm testing, I think, oh, it's a render and error. Oh, they've they've kind of done a hack here, and it hasn't worked.
So I think of it technically. But the way that customers feel about this stuff emotionally is like this, dog **** on the floor of a retail environment. You have no idea how hard it is to find dog **** on the floor of a retail environment in Google, but that's what takes the time in these presentations.
It's the five hours looking for that one image, But that's what it looks like. It says, we put a ******* sign up to tell you there's dog **** on the floor, but none of us are going to pick it up. We're just going to leave it right there, so you have to walk around it.
Right? And that's the impression you give people when you leave these bugs on your website, that you don't give a **** about them. Yeah. Apt. But the call center acts like a giant nappy, absorbing all the customer complaints. Nothing escapes. It's a gravity well.
The only thing that comes out there are KPI sheets. Right? No customer complaints will ever escape the giant nappy, because call centers are designed to bury bury bugs. They either die during the escalation of reporting, and it's rare to find companies who can have a bug hunter approach who are systematically looking for this or create incentives for people to find this stuff.
Yet the effort to find and remove these bugs is phenomenally low compared to the amount of money that you make from fixing them. So the invisible costs to companies are huge. And what most people don't realize is that, you know, Far Eastern car manufacturers beat a lot of Western manufacturers in the sixties and seventies, not because the cars were somehow, you know, super better, but they had less flaws in them.
They didn't go wrong. You had less recalls. The product quality and defects were so much better. My dad's Nissan didn't go wrong in the way that his British Leyland cars did. And so what these guys developed this stuff for is to manage these risks.
Right? And all it boils down to is avoid **** that hurts your company. That's what this stuff is for, and you need the same mindset for your website. So why guess with this stuff? You know, why don't we use our Google Analytics data to answer these kind of questions about device usage on the site?
And over the last five years, our model loads the sites, and this is how to split your Google Analytics data. Don't look at what Google Analytics gives you because it's wrong. Right? It's it's set out wrongly. If you split it in this way, you'll get everything you need.
This model is what you're going to fill in the blanks on. So go here and read this article, which is a fifteen minute model recipe. And there's some groundwork slides in there for analysts, but essentially, it's three steps. You get this custom report from Google Analytics.
You get this Excel sheet that's full of blanks. And then, wow, you copy and paste the stuff or or put those figures into the blanks. And it will take you fifteen minutes, And what it will do is it will start to give you some data about the device and device class performance in your site.
So here's an example. We looked, and we saw that Android devices, see top right here, Android tablet and mobile, are converting half the rate of iPhones. It could be because they have less money. It could be a socio demographic. It's the user behind the device, or it could be a flaw in the device.
This was thirteen bugs that hit all Android devices. Half a day fix. Most iPhones that I see on people's sites, they test big iPhones. They don't test mass market phones like 6s and fives. Right? So what happens is is you design it, and then all the **** that you put in the opening screen doesn't fit on the smaller phones, so they don't convert as well.
That's because you're testing the wrong phone. But if you take all that model I've given you and you fill in the figures here, these are traffic in black across the top, and the figures on the sides are the user conversion rate. This model for a client took three days to test and find all the bugs that this highlighted.
It took one day a dev time to fix, and they haven't spent any more money. Four days of effort to double your conversion rate. Right? That is that is a phenomenally easy win for very little effort. But how do we automate this? We don't want to run these models every week.
So what you really want to do is funnelize this kind of device experience. You want an Apple mobile funnel, an Android funnel, a tablet funnel, and you can do this. You create some metrics, metric groups that you want, and then you can start to dimensionalize these in Google Analytics.
Well, give me those figures for iPhones and Androids in different countries. And I have a tool that will do it for you. This will give you a grid for your site, and you can look along it and find out what's performing and what's not performing.
But you can make your own model for this and make it really unique to your company. You don't require any coding experience to access the Google Analytics API. If you can use ******* Excel, then you can pull stuff from the Google Analytics API data.
These two sheets have all the plug ins and tools and guides that you'll need for Google Sheets or Excel to do all of this stuff. Okay? Two slides. Very, very good stuff on there. So you've done that. You've got the data. What do we do with it?
Well, what one thing would make a difference? Rather than saying that we support devices, why don't you actually ******* prove it? Right? So now that you know which devices are coming to your site, then you can test it, And this is a minimum viable testing list.
This will work for almost every site in Western Europe. All of these crop up all the time. Two mobile devices, two laptops, and two tablets will cover probably eighty five to ninety odd percent of all your traffic. Trust me. I know. Here's a client example.
That's their testing grid. It's an iPhone six s, an s a, a Mac laptop, and a Windows laptop. That covers eighty four percent of all our visitors. And if they want to go a bit higher, they can add two more for ninety two percent.
It's also important to look at screen heights here. If you're always looking at your product through the lens of your giant ******* monitor, it's going to look great, but if you actually dig out the Google Analytics data, and there's a hack in here that'll show you how to do it, you can work out what screen heights people have on their laptops, and you can see this one here.
Nearly sixty percent have less than nine hundred pixels, and this is the problem. So you start like that. Well, nineteen twenty by ten eighty screen can see the whole page, the structure, the add to cart button, but then when you look at the most popular resolution on the site, what the **** has happened to the structure of the page?
It doesn't look like an image handler anymore. There's no add to basket. It doesn't even look like a product page. And it gets worse. This is the most popular resolution on a client I used to have. They've gone bust now, unsurprisingly, but this is their landing experience, which looks like vomit on pavement.
So where'd you get the kit? Right? Basically, you can buy it. Cheap secondhand device lab. Right? It's what you need. So cash converters, Kecks, Amazon, eBay, all these places, there's an entire testing rig that you can buy for less than a thousand pounds.
And you will get ROI on that. You will get the money back because you'll find bugs. And if there are devices you need to test that you don't own, well, you could use the cloud. There's a cloud for desktop browsers. There's two tools on here, but you can also rent real mobile and tablet phones in the cloud connected to the Internet.
If you want to use an iPhone in Rio de Janeiro, it's connected to a telco there and visit the Brazilian app store, you can do this **** now. It's great. So there's no excuse. Whether you buy, rent, make your own device lab, you can do this stuff.
Right? So there's no impediment there. So what did I learn from testing over a hundred products? Well, over ninety five percent of companies test the wrong ****. Their testing list does not match what their customers have and hold in their hands. If your testing list isn't from your analytics data, if it's from someone's head, then it's a lie.
But if you don't test this stuff regularly, people will break it again. The bug can be introduced back onto the site a week later, and you'll never know. And this is an interesting one. We do loads of QA work on AB tests to make sure that the AB tests work correctly on devices, and over sixty percent of those tests fail because somebody screwed something up.
So that might mean that your AB tests are telling you useless information. And the summary of this is that device experience is a c level financial risk. It could cost you more money than your IT budget every year not to get this stuff right, so it's important.
It's like Schrodinger's cat, really. Until you look inside the box, you don't actually know what's happening. It's speculation. And what's the most expensive thing that I ever found? Well, this was a brilliant example. It's called a landing page yield analysis. Like farmers' fields, you look at the crops you sow and how much yield you get back, and this one popped up, the product pages.
Wow. What's going on there? It's like the yield is so low, and it made us go and run some tests on that page. But the conversion through that set of pages after we fixed one bug is nine times higher now. Right? So imagine a share of your revenue of five percent multiplied by nine.
Right? So it's made a huge difference to this company. It's one point three million more sales a month from one bug. Right? And nobody called them. And ninety percent of their marketing spend was wasted, and they never knew. And the quote they gave me was, **** me, Craig.
That simple bug you find is gonna earn us more money than all the freaking AB tests we ran in the last year. And it was true. So stop making assumptions about what devices people have when they visit your site. Make your device model and not somebody else's.
Draw up your testing list that is representative of your customers, not somebody else's. Go and buy the hardware. For goodness' sake, if you don't have this kit around the office for people to use, then they won't serendipitously check things on it. Start testing this stuff.
Get into the discipline of it because there's a cost associated with not doing it. Agile is great. Right? But it's not if that leads to defects in product quality that then kill your business. And I know this last one particularly poignantly because I'm in the over fifties.
Dating game now, nobody will call. Nobody will call. But accessibility is about many more things than I've talked about today, accessibility and inclusion in terms of the traffic that's coming to your site. There are all these things that you need to take care of, not just this one.
But there's a really bad downside to this, because if people can't read, see, the text on your site, or there's contrast issues, or the font is too small, what you're saying is, **** ***, people who wear corrective lenses. We don't want your ******* money.
You can keep it. Great. What about, hey, we don't test on Macs. Oh, we've got someone with a Mac who's on a nice sort of, like, table. Right? **** ***, wealthy Mac owner. We don't want your money. And then what about someone who's over fifty, owns three houses, four classic cars, but happens to have the misfortune of using Internet Explorer eleven? We are saying, **** ***.
We don't want your money, you rich *******. Mean, it gets even better. Let's scale this right up. If you're operating internationally, you probably have forgotten about the countries where there's a large proportion of Android if you're just testing on iPhones in your own market.
So what you're doing now is you're saying, **** *** slightly poorer people all around the ******* world. We don't want your ******* money either. So there's a giant pile of cash on the table that you start with, and gradually, you just start burning it through neglect and not testing this stuff and giving a **** about what customers actually want and what they have in their hand.
And if you look at the losses per month, that's the slide that was from earlier on, which was the amount in hundreds of thousands of pounds that these companies were losing every month from these bugs, and we know that these are accurate because after we fixed the bugs, we got these uplifts, and that's incredible.
You know, it's just such a simple thing, And at the end of the day, are the CEO in charge of this choke point. Right? It's really important that you make sure that everybody can get in here. And if you were running a department store, you'd see it if people can't get in, but you're blind to this because you're running a digital product.
Your eyes come from the measurement. You're effectively blindfolded until you actually look inside the box. So accessibility really expands your reach. It's not like a boring thing that you have to do. Means more people can buy **** off you. It's simple. And device testing itself drives delight in profit.
It makes customers happier. It makes less friction in the world, and it will make you more money if you know what devices to test. So please, take this stuff away, run your own model, and let's all together end device discrimination today. Thank you very much.
