Is the era of Google's search monopoly coming to an end? Tom will outline how Google's search engine, originally designed over 25 years ago for a different internet era, wasn't built to solve the modern demands of search, and the technological advantage they have held for so longer has largely been erased. Tom will explore how competitors, such as Apple, are likely working on alternative approaches to search, for which we are unprepared as an industry. Tom will show how we need to prepare for a short-term future where everything from our existing attribution models through to strategic technology plans need to be revisited.
← All talks 
Grow Turing Fest 2024
Beyond Google: Why 2025 will See The End of Era
Auto-generated transcript - may contain errors. Tap a timestamp to jump the video.
APPLAUSE GERMANS. Germans are a funny lot, aren't they? Got any Germans here today? Quick show of hands, any Germans? Oh, we've got one. Anymore. Don't look so nervous, it's okay. I like Germans. I'm actually married to a German. And a few years ago, I moved from the UK to Germany.
And I was super excited because I was gonna become part of this super efficient society. Germans are very famous for your efficiency. And so after just a couple of days of being in Germany, I was ready for my first taste of this efficiency.
And so I phoned up the doctors and I explained, oh, I'd like to register as a new patient, please. And the lady's like, yeah. Yeah. That's fine. I need this detail and that detail from you. Sorry. Is I forgot to click. This is what German society was looking like in my my mind.
And so the lady on the phone is like, oh yeah, I need this and that detail from you please, can you send it over by fax? I let out a little giggle, didn't know Germans were so funny, sorry. And just silence. I was like, my German's not great. Maybe I misunderstood.
So I better say something. So I'm sorry. I thought you said facts. Yeah. Now say what you want about Germans, but the language at least is efficient. Because that one word was enough for me to know this conversation was over. So I go away and I spend the next hour or so online trying to work out how do you send a fax in this day and age.
And eventually, find a service where I can write something on a piece of paper, and I can take a photo of it with my my phone, and then I can send that to my computer, and I can convert it into a PDF, and then I can upload it to the web website, and I can enter my payment details, and now send the fax for me.
But this was not the efficiency that I've been promised. But the more I got thinking about it over the next couple of days, the more and more it started to make sense. Right? Because just like in the UK, German doctor phone lines are very busy.
They wanna get you off the phone as quick as possible. So they need some way, some mechanism in which you can send them entirely free form data asynchronously in the background, and it will somehow arrive to them, and they can deal with it on their own time.
Fax is the perfect solution for that, right? Until you remember email exists. Email is obviously a more modern, more versatile it's okay sir, you are at the right conference still. I am getting that. Give me a moment, Google, Google. Yeah, fax is a perfect solution, but email is more modern, more versatile, and it's how we'd expect to interact with the world around us.
So like this gentleman, you might be wondering, does all of this have to do with Google? Well, to answer that question, let's start off by having a quick look at the history of Google. So cast your mind back to nineteen ninety nine, this was the first year that I started doing commercial web development work.
And at this point in time, this is what Google looked like. Google logo, text field, a couple of buttons to start off the search, fantastic. You move forward five years, two thousand four when I started doing SEO, Google got a new logo, they've added a couple of links, but it's still mainly talking about this text field and these buttons, another five years, and text field search buttons, another five years, and oh, another new logo, still the same text field, still the same buttons, two thousand fourteen.
Two thousand and nineteen, oh, they've turned the ends of the text field into curves, spiced it up a little bit. But otherwise, I haven't changed much. I've got an icon, we'll talk about that later. Fast forward to twenty twenty four, another five years where we are right now.
In the last five years, all I did was make that text field a bit wider, which is actually a very deliberate decision that we're going to talk about later on. But hasn't much changed. But Tom, Tom, I hear you crying like they've got a mobile site.
What is what it looks like on a mobile site? There's no button because it's on your keyboard. But Tom, Tom, they've got an app. They've really spiced it up there and made the text feel grey. In the last twenty five years, this is the change.
Basically almost nothing other than the logo and the styling has changed about Google's interface. And so we're laughing about Germans, and fax machines, and all that stuff because we consider them to be this product of a bygone era. But Google itself is a product of the 90s, nothing has changed about that primary interface in over twenty five years.
So I'm here today to persuade you that Google was built in a different time to solve a different problem to the modern search problem for a different audience. And I think developments that we've seen in the last couple of years, especially around LLMs, have also significantly weakened Google's position.
So now the time is ripe for others to move in on their space, and I think in the next twelve to eighteen months, we're gonna start to see competitors eat into that monopoly, that choke hold that Google have had. And so I'm going to talk about what that looks like.
I was about to say let's dive in, I haven't spent the first five minutes talking about Germans and fax machines. So we're going to talk through four things today. First is modern search requirements. What should a modern search engine look like? Then I'm going to talk through LLMs leveling the playing field, basically how I think they've diminished the advantage that Google has had for a long time.
Then we are going to look at what might a modern search engine solution look like, and I am going to talk through how I think people are already building them. They are just not what we imagined a search engine might look like. And then I'm going to wrap up with a few final thoughts, a couple of tactical takeaways.
Mostly this is going to be a strategic thought exercise, but I've got a couple of tactical things for you to walk away with. Okay, so let's start off by talking about the evolving demands of search. So hopefully you'll all be very familiar with this typical Google search.
You enter a handful of keywords up here in this text field, and then down here you get links to websites. So the corpus is the web, the corpus being the technical term for the body of content that any particular search engine or search function searches against.
So you've got keywords and web. So already straight away we can make a few observations about Google. The only source of content is the web, there are a few edge cases, but primarily they're mainly searching the web. The results are also purely web based, they're hyperlinks off to websites, and the queries themselves lack a lot of nuance, which is putting in one to four keywords.
This is not a complex query that we're putting in. And the other observation about Google is where they sit in a stack, they're constrained to the browser. What do I mean by that? So Google sits here. You open your phone, if you've got an iPhone, you open Safari and you start a Google search there, or you open your laptop and you open Firefox and you start your search there.
The whole experience is constrained to the browser. And this makes sense if you consider that Google is a product of the nineties, and in the nineties the web browser was the primary portal to using the Internet. You went and you dialled up with your modem, and you opened the browser, and for most people the whole Internet experience was the web.
But that's not the case anymore, nowadays it looks much more like this. You've got the web browser on your phone, but you've also got dozens of different apps that are all potentially connecting to different data sources over the Internet, the device itself is connecting to the Internet in various different ways, etc.
And so my thesis would be that it makes much more sense for search to belong here, search to be part of a device native experience, and being able to search all of your content rather than being one step down constrained to the browser.
So just from these few observations we could already note a few demands of what a modern search engine might look like. The first is, I want to search all of my data, I don't want to be constrained to that browser, I want to look in the apps as well.
The second is that the result format should be native, meaning that if you do a search in a web browser, getting web links makes sense as a result. But if you're searching across all these different data sources on your device, you want the results to be in an appropriate format for wherever that data is.
And then finally, I'd argue that in the last twenty five years, the size of the web, the complexity of the web has exploded, and then you've got all these other data sources, simple one to four keywords aren't necessarily going to be enough in that model to do good searches.
And so, in summary, what I'm saying at this point is that imagine we'd got to twenty twenty four and we'd somehow forgotten to build a search engine, right? What we would build now if any of us were put in charge of this would not look anything like Google.
Okay, so that is the modern demands of a search engine. So let's talk quickly about how LLMs are levelling the playing field. And so, Google I think have got two problems with web search. The first is that for twenty five years we've been trained to search in the way that we do, entering this handful of keywords.
So most of you will be familiar with that. This was a query that I put into Google a couple of years ago when I was trying to plan a vacation, a holiday with my family. And I think most of you would be familiar with this, that this is not everything I needed from the holiday.
I had a bunch of criteria that I knew don't work if I put them in here. And so, this only represented some of those criteria. I wanted a toddler friendly pool, and I wanted it to be near the beach, etc. So what you typically see for most users is you end up doing something like this, you put that query into Google and you do your search, and then you search through the results.
You open up the first five or ten websites, and you look through them applying your own criteria. Is it near the beach? Has it got a toddler friendly pool? Etcetera etcetera etcetera. Essentially, you're doing the second phase of the same search, and rather than Google doing this for you, which should be how a search engine works maybe, you're doing it yourself.
This phenomenon is very well understood, it's called post search browsing. Most of the major search engines have published papers about this phenomenon, so they understand this problem. And then what typically happens at this point is that you do a follow-up search on Google having slightly adjusted those keywords, and then you repeat this process on a loop until you get something approximating what you're actually looking for.
Whereas a much better solution would be able to do a search like this, show me holiday itineraries for a family of five with a toddler friendly pool and near the beach, and some restaurants in or near Europe. But none of us do that because one, we've been trained over decades to enter handful of keywords.
Two, you've got the text box being the size it is. And three, if you ever have tried any of this, Google does that infuriating thing where it just ignores half of the terms and it grays them out like, oh, I didn't really think you gave a **** about that.
And like, well, I've put it in a box for a reason. And so you end up just infuriated and getting poor results. And so rather than do this, you do that process I showed you, and you end up generating something like this. This was the spreadsheet my wife made a couple of years ago for our vacation.
I told you she's German, Germans love a good spreadsheet. I think it's because they're easy to send through a fax machine, I'm not sure. And so you end up doing something like this, but obviously, a Star Trek type modern search engine wouldn't demand that you're doing this.
It would allow you to filter and query inside the search engine to get the results you're looking for. Whereas if you put that same query into something like ChatGPT or another LLM, it understands the query far better than Google does. It understands the context, it understands the nuance, it understands the criteria you've asked of it.
It doesn't necessarily give you very good results because it doesn't have good data sources to provide those results from. However, conceptually it understands what you're asking of. Okay, so the takeaway here is that LLMs, most people when they're talking about LLMs, we talk a lot about Gen AI and all of that stuff, but we don't talk a lot about the fact that LLMs are also really good at helping understand search queries, search intent, that sort of thing.
They're much better at understanding that, whereas Google is very much still keyword centric. And so what's really interesting is Google knows this, Google knows they've got a problem, they know that they're going to be increasingly competing with LLM type models. I think Bethan spoke this morning in the other room, and she she had a slide that mentioned perplexity, I hadn't heard of it, it's a search engine that's LLM powered.
But basically we're seeing this sort of rise. So Google need to retrain us, and so in the last couple of years you might have noticed these filter buttons, these bubble buttons appearing below your search. If you click one of those, what happens is it just literally adds those keywords to your query, and you can click more and more buttons and it just makes the query longer and longer.
I think this is Google trying to incentivize us to change the way we behave, the way we search, because they know that they're gonna be competing with search engines where you do feel comfortable typing in that long complex query. Okay, so the second problem that they have is that LLMs also compete with Google's semantic models.
What do I mean by this? So Google have got this huge web index of all this content on the web, but over the last ten, fifteen years they've also built up other models. So, they have what they call the knowledge graph, which is a huge database of entities, and attributes, and how those things link to one another.
So, you search for Robert Burns, and it understands the entity, it understands these attributes of Robert Burns, when he was born, when he died, understands relationships with his children, So it's got this semantic understanding of a lot of the entities or the popular entities on the web, and this is a huge advantage for Google because it allows them to provide better search results.
And then they enable us to do that with structured markup, so if you're not familiar with structured markup, it is just an HTML extension that doesn't make any visual change to the page, but it's machine readable. So when Googlebot visits your page, it can read the content that is for humans, but it can also read this machine readable content and understand, okay, this is a holiday place, this is the location, this is the number of rooms, etc, etc.
And basically, it allows you to express some semantic structure to that content, and it allows them to do better search results. So schema dot org, which is the name of the vocabulary that Google used for this, it doesn't matter too much what the name is, point is that that structure markup allows you to describe around about fourteen hundred types of entities.
Okay? So that's how Google have been adding this sort of semantic understanding to the structure of the web. GPT-four has an estimated one hundred billion neurons. This is how an LLM stores structure and semantic knowledge about things. And so if it's not clear to you how much of a difference that is, that pixel there is schema dot org, this is GPT-four.
It's literally millions of times the size, I did the math and everything, I remember with PIE and everything, was really proud of myself. And so it's literally millions of times the size, these are two very different things, but they're both designed to store an understanding of the content of the web.
And so you can see there's no competition here, LLMs understand nuance and context far better than Google. And so, LLMs also excel at that semantic understanding of web content, and undercuts twenty years of Google research. Anyone looking to compete against Google now is suddenly a much easier task than it ever was before.
Okay, so I've talked about how Google are constrained to the browser and the modern search engine should be searching beyond the browser, and I've talked about how even in that browser web in like context Google have had their advantages undercut. So, let's put this together and have a look at what a new solution might look like.
I need to speed up a bit because I'm running out of time. So, I'm going talk about Apple, not because they're the only person doing this, but just because they're a simple, a clear use case from my mind. So, what might an Apple search engine look like?
Well, if we revisit this slide, this explains why Apple should care about building a search engine. They want to sell devices, they want you to have a good experience. Once you acknowledge that search should be a device level problem, it needs a device level solution.
So, I wasn't at all surprised to see in twenty twenty Apple quietly made a change where if you asked for you did a swipe right and you did a keyword search on an iPhone, before Siri suggested you websites from Google's index four years ago that switched to Apple's web index.
For at least five years Apple have been crawling the web, building an index, they're already doing this, they've already been working on this. So let's see how Apple might do against our modern search requirements. The first one is I want to search all of my data, not just the web.
Well, Apple have an App Store, it's got two million apps on there, and so you can see that this might be an advantage. Let's have a look at how that might work. You do a search for Madrid on Google, it searches the web, You do a search on this hypothetical Apple device, and it searches the web, but it also asks various apps on your device.
Can you help with this search? And this is a type of what is called federated search, where you ask multiple different actors to work together and each do part of the problem and bring it all together. But how would Apple know which apps to ask?
When you install an app on an iOS device, it can tell the the phone or the iPad or whatever what types of intent it can deal with. So you end up with something like this. You type in this search trip to Madrid, and the first thing is it has to understand what are the intents.
You might be looking to book a flight, or book a hotel, or learn language, or check the weather, or get the travel guide. Once it knows what intents you need to satisfy, then it can start asking the apps on your phone, can you help with this?
And it can ask the web on the side, and it can aggregate all of that together. I have reverse engineered the API that Apple is using for the search results on iOS, and you can see that this is what they're doing, they're blending results from different sources.
So Apple will be able to serve many of the same intents as Google does, without needing to have a web index as strong or as mature as Google's. They just need a web index which is good enough, which they've already got. The second thing about this federated model is that federated results are real time.
If you update your website, it might take Google hours, or days, or weeks to update the search results, Whereas if you're searching on federated model, can go to the app and go straight to the data source and get real time data, what are the very up to date results right in this moment.
Okay. Results should be native. They should make sense given the the format of the data. And so we already saw this, Apple apps can tell you what it'll tell the device what intents they can serve. When they do that, they can also supply a custom UI.
So you ask Siri, I wanna split a check. It doesn't take you into the calculator app. The calculator app knows how to deal with this intent, but it's giving you a custom UI outside of that. And so we extend this, this is because it's one of the Apple's built in apps, but you extend this to all apps and suddenly you've got apps answering your queries right there.
You could imagine this being a list of different results from different apps answering your query. WhatsApp is another example, this is an interesting example because there's also a button here to send, so you're interacting with WhatsApp without ever actually opening the app itself.
And we talked about live results. So you can imagine you arrive in Edinburgh, and you don't know where the International Convention Center is, and so you ask your phone, how do I get there? And it might give you the bus timetable and some other things, and it might also say, oh, but the Uber app says it's an Uber, that's one minute away from me, do want to go on an Uber?
This live results model suddenly is a paradigm shift, it makes this is a search engine, but we've been trained to think of search engines being something that lives in the browser. But how does this work if you've not got the app installed? Because you might ask a question that don't have an app that serves that intent.
Apple have thought of this. In twenty twenty they launched App Clips, which are basically little parts of an app that stream to your device. Nobody seems very aware of them over here, they're quite popular in other places, in Japan it's fairly common for there to be a QR code on vending machines, you scan it and the app opens, you don't go to the app store, you don't download the app, you don't install anything, you're straight into the app experience where you can interact with that vending machine and buy something.
And so basically these are parts of an app designed to solve a specific problem. And so, skip over that, Apple have got two million apps in in the App Store, and you know what intents every one of those apps knows how to deal with.
And so now they've got what I'm calling an intense index. So you do a search and maybe you haven't got the app, but they know what AppClip to serve you to lend to that. Furthermore, once you've got these native format of results, you can start to benefit from the native functions and functionality of the device.
Things like buy with Apple Pay or sign in with Apple. So it's easiest if we put it all together in an end to end example. So imagine I arrived here in Edinburgh, and I'm walking to the conference, and I don't live in Edinburgh, don't know anything, so I do a search, where can I get some coffee?
And my phone says, okay, well there's a coffee shop two hundred meters this way, would you like to order now? Yes. I go straight into the app clip, and I choose my drink, I buy with Apple Pay, I walk into the shop, two minutes later, I pick up my coffee, I have never been to the web, I've never downloaded an app, I've never installed an app, everything happened right there in that search functionality.
And from Google's perspective, they talk about being mobile first, but they're still constrained to the web. This sort of mobile native or device native experience is something I think they'd absolutely kill for. Okay, so you get device native results, I'm going skip over these to save some time, and these can power actions.
But just in the same way as Google needs to change the way that we behave in adding those buttons to make us type in longer queries, Apple also need to retrain us, right? Because this is not the current model, most people do still open the browser to start their searches there.
Interestingly, the next generation are changing their behavior a bit, I have young daughters and they interact with their phones in all sorts of weird and wonderful ways, but they don't have those same preconceptions as we do. But Apple do need to retrain the rest of us.
And so a couple of years ago, March twenty twenty two, I made this mock up and I hypothesized that Apple were going to want to start making us want to incentivize us to do searches right from the home screen. This was my beautiful mock up, A couple of months later they did exactly this, they added this search field to iOS.
If you got an iOS device you are very familiar with this. This is designed to train us to start searches at the device native level. They just announced a few weeks ago that in the upcoming version of iOS coming out in September, you now have to type to Siri rather than have to use your voice, and basically bringing personal assistance and search together into one sort of thing.
We still think of them as separate things, I don't think the next generation necessarily does. Okay. So, if we revisit this slide from earlier where we want to search all of my data, not just the web. This word here is doing a lot of work, all of my data.
If you do a search for I live in Germany now, I need to know how to make good schnitzel, and so if you do a search on Google for schnitzel recipe, you get web results. But you can imagine that in this Apple world I get web results, but in this federated model, there's also my recipe app, and it asks my recipe app, and it's like, yeah, you've got a recipe from your friend Sandra, she gave you a Schnitzel recipe.
This is my data, this is logged in data. Google can't crawl this, it's behind a login wall, but apps have access to this. So suddenly the paradigm shifts again. Now I'm searching actually my data. So I get Sandra Schnitzel recipe, and I also get a message from my father-in-law on Facebook that says, oh the secret is actually to have panko breadcrumbs, or whatever it might be.
But now it's actually really my data that I'm searching. Okay, so this is Brian, one of the founders of Turing. It's a bit of a pixelated image. I did try to upscale it with AI, but suddenly he was Chinese, so I just left it pixelated.
But I was talking to Brian back in April when he kindly asked me if I wanted to speak at Turing, and I had talked about this idea of this shift that we're seeing and how Apple is setting up to do this fully personalized logged in search model.
And then in between then and now, just a few weeks ago, Apple announced that the upcoming iOS was gonna have this Apple intelligence layer that layered through everything. And one of the features of that was something that they called personal context. So I've got a very short movie we're gonna watch quickly.
Next, personal context draws from your photos, calendar events, messages, and other apps. So let's say your mom is coming to visit. Simply ask Siri, when is mom's flight landing? And Siri already knows who mom is and will cross reference flight details from an email to give you an up to date arrival time.
And if you've forgotten what plans you have, you can say What's our dinner plan? And Siri knows you mean you and your mom and will find that information in a text she sent you. So once again, we're already seeing Google acting exactly on the the sort of stuff that we thought they were going to oh, Google, Apple.
Here, it looked in your emails, something that Google can't access because you're logged in, and then it cross referenced that information with real time flight information to bring you an ideal search result. As this expands across more and more apps, it's going to be an absolute game changer.
So, device native results can be fully personalized in a way that browser based results never could be. Okay. So let's wrap all this up quickly. Final few thoughts for tactical takeaways. We should prepare for an increase in dark traffic. If any of you care about search engine traffic, then you should be thinking about how are you going to track this in the new world.
If you do a search currently on iPhone and you look at the Siri suggested websites and you click through, that appears in your analytics as direct traffic, not a search engine traffic. And so we need to make sure that we're aware of this sort of shift, so that when we're tracking all our business intelligence metrics and whatever else is we care about, we should be thinking about this.
If you've got an app, you should also be thinking about your app analytics if you're not already. Any content we write should be designed to feed LLMs. And so, if you think about that example of that holiday search I was doing earlier on, I was asking about toddler friendly pools, and is it near the beach and all these other things.
You should make sure that if you are that company company selling holiday villas, that you're addressing all of these points so that the LLMs can learn off of all of this. And there's definitely a first mover advantage here as companies move towards doing more LLMs.
It's estimated that ChatGPT four, I think takes seven months to train. So, you should be updating your websites in time for the next round of training to take place as these models become more prevalent. We've talked about how I think search belongs at a higher level than just a browser, it should be as this device native experience, I think that's critical.
But web search is still going to be a big part of the pie, and that is becoming more and more complex. As you've got LLMs understanding queries better, you've got LLMs understanding your content better, it's become harder and harder than ever before. I'd be remiss if I didn't point out that I'm CTO of SearchPilot, where we do SEO AB testing to enable you to make data driven decisions for SEO.
We built SearchPilot on the belief that web search was gonna become more and more complicated and it was being impossible to predict. I've got two seconds left, so I've got two more slides. If you don't remember anything else about the presentation today, you should update your mental model of what a search engine is.
We've all conditioned to believe it's this thing that belongs in a browser that is a web focused, but anyone designing a modern search engine would come at it very differently, and as I've shown you Apple already are, there are others doing the same.
For the last twenty years, we've treated Google as being this untouchable, unbeatable thing, but all of the changes of it moving out of the web, all of the changes of LLMs diminishing their advantages means they're weaker than they've ever been before. So, I don't need to have been right about all the details today, I just need to be directionally right that this sort of thing is happening, and then any future does have a far more diverse set of traffic coming to all of your websites.
Thank you so much for your time. This is me on Twitter if you want to talk to me. If you're German, my fax number is there. Thank you very much for your time Edinburgh.