This talk is an overview of what Fergal has learned at Intercom, through two product development cycles using GPT-3.5 and GPT-4. Fergal will try to equip the audience with some mental models to think about the capabilities of GPTs, and share tips for how to build with them, based on Intercom's recent experience, as well as some general thoughts about where we're going next.
← All talks 
Build Turing Fest 2023
Building products in the Age of AI
Auto-generated transcript - may contain errors. Tap a timestamp to jump the video.
Everybody. Thank you for coming to my talk here. I hope you get something useful from us. My name is Fergal Reed as James said. I work with a company called Intercom, kind of B2B SaaS. We've recently over the last six months really been through two product development cycles using GPT, and I wanted to come here today to share a little bit about what we've learned doing that, and maybe also share some mental models that we've developed as we've gone that we've found useful for thinking about this sort of technology, how to use it,
how to put it in production. And I guess to start out by setting out my stall here, really believe GPT and LLMs or whatever you want to call it, is sort of an internet size change in the technology landscape. This is a big deal.
This is a new technology capability that we haven't had before, and it really changes how we build and how we use AI right over our entire software development process, and so I really want to try and convince you of that today as well as give you some kind of tools and tactics that we use to work with this technology.
I thought it would be fun to start out by trying to introduce myself, so I went to GPT-four and I kind of said, hey, what's Fergal Reed's biography? It gave me some things that were quite right and quite accurate and very specific, so it's definitely talking about me because it refers to a paper I wrote, but uh-oh, there's a lot of things in there that are in fact not right and that are hallucinations that have been made up.
I really think we all sort of went through this journey over the last sort of six months. We also kind of chat GPT and kind of the level one kind of thinking in the journey is, wow these things are incredible, they know everything, they're going to change everything.
Then we got to level two, which is GPTs make things up and aren't trustworthy. I guess really what I want to tell you today is that I really think there's a level three, which is GPTs, they really can be incredible when used right.
I want to give a little bit of opinion about how to get there or how we feel we've gotten there. I think the number one thing is to see these things as engineering components. A lot of us were really blown away by ChatGPT.
It was amazing. We played with it, But we were using a specific product with a specific set of product trade offs. When you actually instead look at this tech as engineering components, you can treat them differently and you can get something different out of there.
Really this is about separating out aspects of these systems that were accidentally, in my opinion, bundled together due to how they are created. So let me unpack that a little bit. Let me talk about what is GPT. Now I think we all probably are aware that these are big systems that you give them some words and they try and predict which words come next, right?
They're trained to predict tokens. They're trained to predict what's the next word that's coming in the sentence. If you say something like the quick brown fox, it's trained on a lot of data from the internet and it's built this model and it's smart enough to say jumps over to lazy dog.
Start giving it a sequence of numbers like Fibonacci. It's seen this a lot on the internet, so it kind of completes the sequence. But then something interesting happens. If you say the smallest household item too big to fit in a suitcase is a microwave.
However, this could also vary depending on the size of the suitcase and individual preferences. Order items that are too large or small kitchen appliances, like toasters, slow cookers or air fryers. Something just happened here. There's something different and special going on here. To complete this sentence, I Googled and wasn't previously out there on the internet, the model had to understand something about the world, You need to understand something about the relative size of a microwave and a suitcase, and the kind of the uncertainty there, the kind of the different candidates,
that's pretty impressive. That demonstrates some understanding. You can think about these things deep down as like a sequence model, maybe you've heard it uses attention and it's trained with gradient descent, but I'm going to say that that model might be useful sometimes, but it's probably not the right level to start with.
That's probably not the right way to think about these systems. That would be like thinking about a human according to our genes and evolution. A human is like something that evolves, a bunch of genes and natural selection in millions of years and you get a human.
I guess at one level that's technically true. That's not a useful way to predict what a human is going to do in any given day. So I kind of say with these systems you should sort of distrust any argument that says, oh well, it definitely can't do something because it's just trained to predict the next word.
That's kind of not useful for practical purposes. What really, really interested here is the intelligence of the system and how we use that. I'd really say that a better way of thinking about these systems here is as like a database and a reasoning engine.
I think that's a better model. I would say that these two capabilities, the fact that these large language models, that they know things, that ChatGPT knows certain facts, but then also that it can reason, they really got bundled together because of how it was trained.
I would say that that's almost an accidental property. As a builder wanting to work with these systems, it's useful to try and separate these two things out. I would really say that the reasoning engine and the reasoning capability that we have now is what's key.
That's the key advance here for a lot of industrial applications. Often the database, the things, the facts that the system knows is actually a liability because it's inaccurate, because it's been described as a blurry JPEG of the web. It's blurry, it's approximate, and if you try and query that database you may get information back that's valid, or you may get information back like we saw in my intro there that's made up, that's hallucinated, that's plausible but wrong.
So in a lot of industrial contexts you actually want to try and end run around using that database at all and just use the reasoning engine. Again, it takes a while to get used to this. I'm not going to do a whole talk of just looking at GPT prompts, but want to show you another example that I think is quite compelling.
A mouse wants to steal a piece of cheese. There's a cat in the room, there's a bell in the room. What should the mouse do? The system here comes up with a plan, right? Just with that domain that we've just sketched there, wait for the cat to sleep, move quietly, avoid the bell, etcetera.
Given a new fact, the cat is deaf and it tells you to approach the cheese cautiously. What about the bell? Ignore the bell, reason the cat is deaf. The bell sounds won't alert us. Unless you've worked in machine learning for a while, that might not seem as amazing as it is.
People struggled for decades to try and get systems that you could give a real world context like that to, and that could plan in a real world context without a combinatorial explosion or without doing something that didn't have common sense. People for a long time were like, hey, machine learning's good, but it doesn't have common sense.
I think there's a sense in which these models now give us that capability that we were looking for decades and that we didn't have. I think that now can be built into products when you approach it the right way. I think one model that we find interesting to think about, okay, how do you actually use this thing as ignoring its database?
How do you just use its reasoning capabilities? As we talk about interpretative versus extrapolative tasks. What does that mean? You want to give it something where it has to fill in the blanks for. You're not trying to ask it to predict something that's way outside its experience.
Here's a graph of interpolation versus extrapolation, which may be familiar, know, maybe not. Interpolation is like you've got training data of any sort of machine learning system, You're like you're saying, hey, similar to the points you've seen or similar to context you're familiar with, try and make a prediction.
Whereas an extrapolative task is like, hey, something you haven't seen before, try and predict. It's way outside your experience. You're really trying to push these large language models to do things that are at some intuitive level more interpolative for them. As an aside, I went looking for a graph like that, which is a graph you'll find in a lot of science or machine learning textbooks, and I couldn't find what I was looking for in a minute or two of Google, and it weirdly turned out that the fastest way to get that
graph was to say to GPT-four, hey I'm looking for a graph of interpolation versus extrapolation, generate me the Python code to write that graph. That's one shot. That was a single request to GPT-four for the sort of graph I wanted. It generated the Python code, the plot of that graph, and map plotlib.
Again, just think there's this total change in capabilities here when the fastest way to get the graph I'm looking for is to query a large language model that produces it rather than search for it on Google. But anyway, these things are less reliable at extrapolation.
So don't form your query as who's the president of the United States in nineteen oh two or something. Instead say, given this article about US presidents, who's the president of the United States? Favour interpolation. Set these systems up to perform a task given a context.
One example of this is something called retrieval augmented generation. I would say that this makes sense at a human level, and a lot of the debate around these large language models misses that humans do this too. If you have a human who's like, maybe they're a history professor, they're an expert in a particular area, if you ask them to answer a deep enough, specific enough question that they haven't thought about for a while, you force them to answer, they may give you something that's plausible but wrong.
Really, if you want to get very high accuracy from a human expert, you tell them to go look it up. You give them a history book and then you ask them the question. Really, this is kind of what you want to be doing when you're using these large language models in a lot of industrial contexts.
You don't want to just stand up your large language model and have people go and query it. You kind of want it to use it as like a reasoning part of a bigger system, where the bigger system is kind of trying to go and do the equivalent of giving it a history book to read from.
From an engineering point of view, that's probably what you want because then you can control the knowledge that the system uses and you can manage it and you can update it, you don't have to retrain the model. I really think that's kind of a first engineering breakthrough to using these systems effectively.
When you use them like this and you frame them like this, a lot of the criticisms in the wider discourse around them kind of goes away. It doesn't really matter so much that the database bundled in the large language model happens to have a fact in it that's approximate or is out of date because you're always using them in conjunction with your existing database, your existing knowledge graph or whatever.
When you use that, a lot of the things that people are talking about as problems kind of goes away. You know, can only pass them a certain amount of context. You can only pass them maybe thousands of words. You can't really put your whole knowledge base or context in it, What a lot of folks are doing now is they're marrying this with vector search.
Want to ask a question of a large language model. What you do is you go and you use vector search or more traditional search to try and shortlist a whole lot of documents that contain the terms in the question that you want to answer, pass them into the LLM, and then say, okay, given all these documents, what's the answer to my question?
That feels like a really, really good paradigm. It's something we've been using. Okay, so that's kind of like selling out my stall in terms of like, hey, there's a discourse around these models. Some of it I think is kind of still at that, in my opinion, level too, and I kind of think that when you actually start just treating them as engineering components, you can go further on and you can pass by a lot of things people feel are stumbling blocks at the moment.
Now I want to switch gears and talk a little bit about our experience at Intercom, how we have built with these models and what's that looked like for us. Before I do that, I want to give a little bit of context on Intercom.
Intercom, we're a B2B SaaS tool. We make this product called a messenger. These are pictures of some of the messengers around the internet, so someone can go and click on the messenger and talk to customer support. We make this inbox for customer support reps to be able to go and answer support questions people have.
Then finally we have some experience building bots, building last generation technology bots, bots that use NLP and medium sized neural networks and intent detection and so on. Recently, if you go to our website, see we've really almost pivoted to be AI first, that's kind of reflected in our branding and so on.
Because we really just think that customer support is going to be completely changed by this. Customer support for a while has been ripe for AI technology. There's so much structure, there's so many people answering similar questions again and again and again, they're answering it with text.
These new capabilities in large language models were just a really obvious fit for the area that we operate in. We've kind of been expecting something in this space for a while. We were reading Google's papers, Google's Lambda paper and so on, and we're like wow, this is very impressive.
If we could get one of our bots to dialogue like that, that could be a game changer for customer support. We're starting to do some work in our inbox, experimenting with generative models, and then thirtieth of November happened and ChatGPT came along and everyone was on the company's Slack late at night saying, wow, this is amazing.
This is very impressive. OpenAI at the same time shipped a model, a kind of a back end model, Text DaVinci three, which is like an API version of similar tech to ChatGPT. We got our hands on that and we were like, wow, we can build products with this.
We should give this a go. And as always, internally, there's always a discussion. There's always like, okay, so is this real? Is this like, are we missing something here? Is this really as impressive as it seems like? Can we build products from this?
How's this going to work? So we set out to build some features, really early, fast experimental features to try and understand, okay, do customers want this? Will this really solve people's problems? Are we getting caught up in the hype? Because when you're in the middle of the hype it's very hard to know, you can't tell.
So we set out to build a set of features, and we set out to build pretty simple features, and I'll talk you through our choice of those in a minute. We set out to build features just for the support agent in the inbox.
Summarization and then the ability to edit the tone of voice of something you wrote, and the ability to try and write shorthand, and then kind of expand that shorthand out. We did this over, in our opinion, quite a fast timeframe. On fifth of December, we were like, okay, we've identified some features, we're just going to give these a try.
We started to build them. Before the holidays, we had them running live for our own CS team, and then in January we got them to customers, and then at the end of January we did a launch publicly with these features of testimonials and quickly started getting customers using those at scale.
Maybe we had hundreds or maybe even thousands of customers using those by the end of February, something like that. So we thought this was pretty fast and we worked **** ** this. How do we manage to do that? Well, lot of how we manage to do it is I think explained by this model, which is that some AI features are very easy and you can build quite quickly in your product with these large language models, whereas some are hard.
I know it's kind of a simple model. But basically, what makes a feature easy? Well, is easy to build with this new tech if the out of the box accuracy you get is high, and if you can put it in a product context so that the cost of an error is low.
We kind of looked at this. One of the first things we kind of came up with is this task of like draft me a summary of a conversation that I've just been looking at. Because support reps often before they hand off a conversation to someone else, they have to write a summary and a lot of workflows, and that takes time.
We tried to build this about a year and a half ago using BERT and T5 and then pretty modern neural networks. They just couldn't understand the flow of a conversation. Human conversation is quite complex and quite hard to get natural language processing to work well for.
But with the new models, with GPT three point five, this worked almost out of the box, and so very quickly we were able to build it. I think this feature choice, there's a little bit of subtlety to the choice in this feature that I think it's worth looking at.
It's draft me a summary. It's not summarise the conversation. It's like write a draft for something the human was going to have to do anyway. The human can then, they approve it, they get to edit it, so the risk of an error is a little bit lower.
That was kind of a good, somewhat conservative feature that we could ship and bring to production pretty fast. Whereas a relatively harder feature, while the out of the box accuracy would be low, the cost of an error might be high. So for example, a bot that answers people's questions.
We got there in the end, but we wouldn't have gotten there in that timeframe. Again, summarisation, it's kind of an interpretative feature. It's like take this conversation and compress it down, make it shorter, write a summary of it. These models are good at those things.
You can't ask them to predict what the teammate's going to say next. That's a little bit harder. Development tactics. Anytime we build machine learning in Intercom, we always have a playbook that we try and follow. I think it's just worth going into this a little bit.
We always optimise for fast customer context. Now, Intercom has got R and D principles that cut across everything we build, which emphasise this, starting with the problem, shipping fast, but it's unusual or somewhat unusual to have an ML team that is so focused on this.
We really believe we almost measure the success of our products by how fast we can put something in production, anything in production, even if it doesn't have any ML in it. So that's a little bit unintuitive, but we think when you're building machine learning features, you really just need to get customer contact trying to engage with the problem, even if you don't use any machine learning as fast as you can.
Then you kind of like AB test your way into a place where you're using machine learning to do it. I think that these new LLMs make that even easier than it used to be. These days most machine learning features, or certainly a lot of machine learning features that you would want to build in a SAS context, you can almost build version one of them just by using a really powerful, really expensive large language model.
We've really found ourselves in times where we're like, oh, we need a classifier to do some task. We can do this really easily with a cheap classifier that we could have wrote for years, but it's even faster to just tell GPT-four in English the task we're trying to do, put that in production, and then we replace it with a classifier later if it turns out anybody actually wants it and uses it.
So we really think that there's a new way of using these things to get much faster speed to production where you always start out using a totally overpowered, totally expensive general purpose model like GPT-four, put that in production and use that to almost prototype the full machine learning system you're building.
People often say, language models aren't all of AI. But there's a sense in which they are now, I would almost say, because number one, they're so powerful and just such a big thing in this field and they can do so many things we couldn't do before.
But also how we build software has changed. I want to talk a little bit about a harder feature we built, which is a bot we made called Fin. As an example of something that's not so easy to build, it took us some time, and as maybe an interesting case study on generation two feature for us.
We built this between February and still building it, although it is generally available now for about the last two or three weeks. This is a product where it's an actual bot we built to try and use these large language models to answer customer questions.
As I mentioned before, we had bots that were previously in production that worked pretty well. People could go and resolve maybe thirty percent, forty percent, something like that of their customer support queries if they set the bot up and if they did all this work to curate intents and to train the machine learning system.
Some customers went and did that whole journey, but there's a lot of friction there, so a lot of customers didn't. We saw with these new GPT models, we thought, okay, it's possible to build a next generation customer support bot here that uses the power of these large language models to try and understand, like I mentioned before in the inbox, these things they can understand human dialogue in a much better way than anything we could build before.
We can get that level of understanding. But also hopefully they could do that without requiring a setup. So really the design goals for this bot that we ended up calling Finn, we wanted it to be able to talk naturally with people. We wanted to be able to run without requiring much setup.
Then were pretty sure we could do those two things, but could we do it like reducing the hallucinations, reducing the times of making things up when it isn't sure of something? Could we do it using just the business' information, which is very similar to the reducing hallucinations?
Our customers probably didn't want the bot that could answer questions based on all the information that was on the internet a year ago, right, because we'll be talking about things that they didn't want their business to talk about potentially. This is reputational risk.
Again, using that technique I mentioned before, so like underneath the hood, and this is an emerging paradigm I think, surely other folks are working on these things as well and there's open source libraries that are starting to do this. When you ask this bot a question, what we'll do is we'll go and we'll search all your knowledge base, all the other content you've given us, the contents of your website if you've given us that.
We'll search that with a more traditional vector search, kind of a neural network powered search, and what we'll do is we'll shortlist all the relevant content that we think might have the answer to the user's question. We take all of that, we pass that into GPT-four and we ask it to resolve the query.
That whole system end to end works. That's an example, I think a case study of a product that we have been trying to build in this direction for years and then suddenly in a matter of about a quarter, what a big effort right across R and D team, Intercom's R and D team, We've managed to ship something that does this that we think could be transformational.
It's an interesting example of something that we have struggled to build or tried to build for years. It's just a big change in capability, a breakthrough product we couldn't have built before. It's very expensive for us to run. We're using GPT-four in production at the moment.
We're trying to figure out how to cache that. It's pretty tricky to cache the highest language models because you don't want to cache like PII and you don't have a lot of control over what they do. Overall we're very happy with it and we're working to try and get the cost down to figure out when we need a powerful model, when we can use a cheaper model.
I think that this is the game a lot for a lot of LLM powered products. You can ship something fast, and I would encourage people to do so, get in market with the biggest most expensive models and then figure out how to make them cheaper and make them more efficient to run over time.
This feels like the new game with this sort of like using LLMs for machine learning is you can get to production quite fast, but then you take on this dash of having to scale them and make them efficient. I guess a last point on that.
In LLM, there's the wider debate of it. They can seem inert. It seems like they just complete tokens, but they can easily be turned into an agent. I think one thing that's interesting about that bot we built is we used the large language model, as I've described, to have it actually answer questions, but also we use the large language model to drive the outer logic of the bot.
When the bot decides whether it should search a knowledge base, or whether it should ask a user for confirmation, or ask a user to disambiguate question and so on. We're using the large language model to drive that too. There's a sense in which traditional software gets turned inside out a little bit here.
I think that's a capability we're going to see again and again and again. I guess my key points of the talk so far is we really feel this is an internet size change in capability. It's big. Internally I guess some people feel it's more of a mobile size change.
I kind of say an internet size change. I think a good model to work with these systems is either a database and a reasoning engine, treat the two separately, don't get confused. I think this changes how we build machine learning, but feature difficulty varies from feature to feature.
For the last short part of my talk I just want to give some guesses about what's next. I'm being transparent, this is guesses. When we were working quite closely we got access to large language models from a number of different providers. They're very impressive.
So maybe I'll just share some thought on how we see this space playing out over maybe the next year. I really think that as SAS companies adopt these features, we're on sort of a maturity journey. Like version one, right, is kind of text tools.
Hey, I've got like a text box. Users type stuff in. I want to make it easier for them to do that. So it's kind of like auto completion version ten or whatever, much better auto completion than we've ever had before. And working around clunky interfaces.
A lot of the times when there's like a task I do rarely that has a big complex interface I have to use, maybe I can just describe what I want to do in my native language And then the system will do it for me.
That feels like version one. Version two is like features for that reasoning capability which you can isolate and treat to the system tool, features that can enhance. I will almost say that our bot fin is an example of that. It's not just this is a text field in my product that I'm plugging an LLM into.
It's a feature or a product built from the ground up using LLMs as a building blocks. I really think that to get access to their reasoning ability. Maybe where this is going next is end to end problems, end to end business problems, customer problems, a task a customer is trying to do where intelligence can kind of help that task end to end.
Again, don't underestimate the reasoning capability of these models. It's very sophisticated. You can set the problem right for them. They can make a lot of progress on it. Zooming out a little bit, I really would say it's easy to lose sight of just how breakneck the progress in this field has been.
When we compare, we were early beta users of GPT-three, not GPT-3.5 maybe two years ago. We played with it and we didn't feel it was ready for adoption yet for our sort of needs. When you compare where that was versus six months ago with the launch of GPT-3.5, ChatGPT, versus GPT four and now, that progress has been extremely fast.
The reasoning has gotten amazing extremely quickly, and so the progress has been breakneck. There's a lot of discussion at the moment around smaller models and open source models. Is the future smaller models rather than these huge big behemoths like GPT-four? I would say it's very exciting.
Some of the discourse around open source models, they're as good as GPT three point five turbo and things like that. Some of that is kind of overhyped. We've tested almost all the kind of open and quasi open models we've been able to get our hands on.
And they're not, in our opinion, they're not as good, and certainly for our tasks are not as good as GPT three point five is yet. So very exciting to see open source and more open kind of versions coming along, but maybe slightly overhyped today.
Productisation. Like I've kind of hopefully tried to touch on in this talk, we certainly went on this journey to understanding, okay, here's the capability, now how do we put it in a product? I think we're all going to get better at that. We're all going to get better at learning.
Now we've this new material as my designer friends call us, new material to work with. How do we actually get good at working with that? I think we're going to see a lot of improvement in productisation. It's going to get everything I just described that we've kind of worked on.
We were early in so we had to swash and do stuff ourselves. But I kind of think it's, you know, reality is a lot of these things get easier and easier and easier. This stuff is so broad and so horizontal that I think open source libraries and common libraries would become better and better building blocks.
I guess the last thing I would say is as I look out and try and speculate on the future, I think smaller models are going be very exciting, but also I think that there's probably room at the top here. I think larger models are probably going to continue to do exciting things.
I know there's been different discourse about it, but bet we're going to see folks continue to train bigger and bigger models that get smarter and smarter and more and more powerful. Again for us the idea that you can take this general purpose model and we can give it the knowledge base of any one of our customers and it can fairly intelligently answer questions, not hallucinating very much at all, really feels almost human level to us.
That's just huge. That's just something that wasn't there and didn't exist in the world even twelve months ago. I think it's very exciting and it's going to be very exciting to see what larger models do. I certainly wouldn't bet that they won't continue to improve over the next few years in very exciting ways.
I would encourage everybody who isn't using these things to give it a go. They're much easier to get started with than you might think. They're a very powerful tool to build products and we've managed to build some things that we couldn't a year ago.
So thank you very much, everybody. Hope that was helpful. Thank you. Awesome. Thank you very much, Virgil. That was a real tour de force. As well as all of the