We now hold conversations with our devices as if they were people - but how did we actually get from counting word frequencies to autonomous AI agents that can take action on our behalf? And once you understand how large language models really work, how do you get good results out of them?
Jonny Brooks-Bartlett, a senior machine learning engineer at Spotify, traces the milestones that led here - bag-of-words models, word embeddings, the attention mechanism, and the transformer that powers today's LLMs - before turning to hands-on advice. He explains why LLMs are just next-word prediction machines (and hopeless at maths without tools), what makes an agent different, and practical prompting tips around personas, output formats, examples, and the real trade-offs of tokens, latency and cost.
Auto-generated transcript - may contain errors.
Tap a timestamp to jump the video.
Hello. Hello. Thank you so much. Thank you for being here. There's other places you could be and you're here for me. Fantastic. And also thank you to Brian as well for inviting me to talk because I quite like to talk. I can talk for ages and, you know, it's a challenge to keep within the time, so I'm gonna try.
But, yeah, this is this is right up my street. Yeah. For for those that don't know, I'm Johnny, and I am a senior machine learning engineer at Spotify. And I've been there for about four and a half years. I work on Spotify search.
So, you know, if you type stuff in and you don't find what you want, please don't complain. But yeah. But today, I'm not gonna talk about Spotify search. I did that last time I was here. I wanted to talk about something else. And, of course, all the hype, gen AI, AI, LLMs, let's do it. So I was like, I've gotta talk about AI.
Gotta talk about LLMs. And in particular, I was like, okay, if I'm gonna talk about LLMs, what is it that I wanna get across? You know, what's the cool thing? So I thought I wanted to, like, first start off and, like, just indulge in, like, my academic geekiness, and, like, talk about how we got to where we are today.
Because when you think about it, like, it's nuts. Right? We can literally have conversations on any device that we have, like, we're talking to other humans. It's fantastic. It's insane. But how do we get to this point? So I wanna start off talking about that.
And then, you know, I I I can't just indulge in academic geekiness the whole time, so then I wanna give some, like, practical tips with LLMs, working with them, prompting, and maybe building applications with them. And then I wanna finish off if I get time, if I haven't spoken for too long, we'll see.
Then I'll talk a little bit about how I use LLMs, which I think just fun, you know. So let's go do that. Let's do that. So here's a little timeline about sort of like understanding natural language, and I've just picked some milestones. Right?
Like, I've you know, if you ask someone else, maybe they'll have different milestones as to how we got here. But, you know, this starts off, like, pre two thousand tens. What were we doing with language pre two thousand tens? And how did we get to a point where in late twenty twenty two, we released ChatGBT, and we have these agents that are now, like, the you know, we're all talking about agents.
That's what all these talks are about, like, essentially. So how did we get here, and what are they? So first thing is we keep using the term LLM. But how many people can actually tell me what a language model is? Like, it's it's not clear. What is a language model?
So let's start with that. Well, a language model is simply just here's some words, and I'm gonna attach a value to those words, typically a probability value. So in an academic sense, a language model is a probability distribution over tokens. And here's a language model.
Here is one. I have a set of words, and I have a value attached to it. It's not that great because these values are static. They but I could, using these, generate a sequence of words. I could generate what chat GBT does. It will look terrible, but I could do that with this. Right?
So all it does, if I have these probability values, I can predict the next word that will come. I can tell you precisely what word should come next. And this is ultimately what we did pre two thousand tens, essentially. We took took lots and lots of text, lots of documents, and statistically went through, and we'll we'll get we'll get to an example.
But we typically call this bag of words. So anyone in the sort of natural language processing space will know this as NLP or or as, yeah, bag of words. It's just, let's take a set of words and assign a number to them. That's what we did.
So, let's look at the the this two phrases at the top here. We have the cat sat on the mat, and the dog chased the cat. Right? So, what I can do with that is I could just count the number of times that the words appear, put them in a in a table, and then divide through by the number of times that I see those words.
And there we have a pro a language model. That's it. From two documents, I have a language model. It's terrible, but I have one. For those who are keen eyed, who are hawk eyed, and are very observant, you will notice that I've made a mistake in the calculation here.
You'll notice that the doesn't appear three times. It actually appears four times in those two sentences. And for those who are even more observant, will have noticed that on the opening slide, some of the words didn't actually make sense. They weren't real words.
So that will give you a hint that I made these slides with an LLM. And actually, this is the first thing to know. LLMs, they're just prediction machines. They just predict the next word, and they can create tables and images and all that stuff.
They are bad at counting. They are bad at doing maths. If you want to do some numerical stuff, you wanna do counting, don't use an LLM. Don't do it vanilla. You want an agent to do it for you. You want an LLM that has access to tools, hence why we talk about MCP so much.
So I'll talk a little bit about that later. But LLMs are bad at doing maths. That's one thing to take away. Okay? Awesome. So let's think about this. Actually, the first thing before we go into the limitations, right, this seems really basic, but we could do some really powerful things with this.
Right? If all I have are word frequencies, I can classify documents. When I worked at Deliveroo, we were doing menu classification. I know that if I looked at a frequency of fajita in menus, I could probably classify it as Mexican. So we could do things like document classification with with simple language models.
That's one of the few things we could do. So it was still powerful even though it's very basic. But when we want to talk about natural language understanding or NLU, this still has some limitations. Right? This bag of words things. Right? It lacks context.
We don't know. I put a table of words and some probability values, but those it doesn't tell me, like, what sequence it was, what word is should most likely come after the word the. It doesn't give me that. It lacks context. It also lacks semantic understanding.
If I only give one number to a word, it knows that it turns up with this probability, but what does it mean? What does cat mean? Is that related to dog? I don't have that. So I don't have a lot of information about it.
And I get, you know, if I have lots of documents, sometimes words don't appear that much, so there's sparsity issues and things like that. So, basically, what we've done over the last fifteen years is address these issues and we get to chat GBT.
So let's talk about those. Let's let's get some milestones that allowed us to address some of these. Right. So the first the first one, two thousand thirteen, paper released by Google. Some of you may have heard of word2vec or word embeddings. This is the paper that started it. Right?
Basically, what they said was, let's not just assign one value to a number. What about if I assign a set of values to a number? Right? And what about if I find that set of values such that it has some meaning in space of some sort?
So what do I mean by that? So here we've got, like, queen and king, which is in the image at the top. King spelled incorrectly. Yep. LLMs, they're not very good at creating images that give words that are spelled correctly. But what about if I have queen and king?
Well, maybe if I look at, like, one number that gives the dimensionality along royalty, those two values might be the same or or very similar because queen and king are are royalty. But when we look at maybe gender, for example, they might be on opposite sides of the scale.
Right? So we can choose a set of numbers that give some meaning for different words, and that's what we call a word vector. And in fact, we don't just have to apply it to words. We can apply it to all entities, like most of the companies, including mine, will have you.
You as an individual, as an entity that we assign a set of values to, and we can do some personalization. So be happy. You guys all have your own vector. You have many vectors. Google might have one on you. YouTube, Facebook, Spotify, you are all just lists of numbers.
Fantastic. But But if you are a list of numbers, we can do lots of mathematical operations. We can do things to work out how similar are you in space. So we do this thing called a cosine similarity. That's how we work out whether you might want this recommendation at someone else who's similar to you once, or that's how I might work out that words are similar.
And I might work out that a word might be placed after another one because I know words with this meaning might appear after a word with a certain meaning. So this was the start in twenty thirteen of us understanding words, and it's still used in the LLMs.
They're just word embeddings. So that's a fantastic. Then we move to twenty fourteen. Right? We understand individual words, but we also need to understand how they relate in context. And so you may have heard of, may or may not, my academic thing, the attention mechanism.
This is where we try and understand within a sentence how to put some words that have meaning together. So let's take the example here. Right? The animal didn't cross the street because it was too scared. Right? If I said that to you, you immediately can know that when I talk about when I say the word it, it refers to the animal.
How do you get a computer to do that? That's very difficult. Right? So we needed a way to do that, and it wasn't until twenty fourteen that we came up with this attention mechanism. And this is what allowed us to then say, okay, if I look at the word it, I can assign greater weight to the other words, the animal, that are related to this.
And this was a huge breakthrough in translation. So this is why a lot of translation apps and tools came bigger in the mid two thousand tens, because of the attention mechanism. We can now take a language, and we could make sure that when I translated it, we could weight the right words in the one language with the correct words of the other.
And so we could start translating things even more. So now we have a better understanding of words, you know, the semantic, and we know what other words are important to it within a sentence. So we're getting closer. We're getting closer to Jack GBT guys. Getting closer.
This is twenty fourteen. And then the game changer, twenty seventeen comes along. Twenty seventeen. We get the transformer. Right? We stack a bunch of attention blocks together. We put on some more additional neural network stuff and or neural network magic, shall I say, and we get something called a a transformer.
Now, you guys, we everyone's using the word chat GPT. GPT is an acronym for generative pre trained transformer. So as you can probably imagine, a lot of the LLMs, almost actually, every LLM uses this, uses a transformer of some sort. They'll have different ways to put it together, but it uses a transformer.
And essentially, this is incredibly important because one, it allows us to extend to much, much, much longer context windows because I apply a lot of attention blocks. And remember, the attention blocks allowed me to work out which words are important. So, I put a lot of them together, and now I can get even even longer context windows as to what words are important to the next one.
And it allows us to process things in parallel. And this is huge because up until now, when we read and when all of our mechanisms we talked about before, we would do it in sequence, you know, from left to right, or if you depend on a language, you might read right to left.
But it was still in sequence. Here, a transformer allows us to process things in parallel. Right? This is what makes us be able to process a lot of text really, really quickly. Right? You think you can, like, just upload a PDF to chat GBT, and you still get responses in roughly the same amount of time.
That's because it's been able to be input in parallel. So the transformer was an absolute game changer. And so, yeah, we get to well, twenty eighteen was the first GPT, that was GPT one. Twenty nineteen, I believe, was GPT two, And then rather than just call chat g b chat gbt three, we go to chat gbt, and that was end of twenty twenty two.
So we get this, and this was huge. So you just stick a bunch of transformers together, and you get like chat g b t, or at least a pre trained version. But one thing they did on top of that so remember I said, LLMs, all they do is predict the next word. Right?
That's all they do, predict the next word. But you'll notice that when you type into chat gbt, it gives a lot of words, and it also follows your instructions. So chat gbt, along with a lot of other LLM models, they are further trained.
So it does not just train on the web, right, so it gets a lot of information from the web, but then we do something called instruction tuning. So that is where you now give the model some a set of instructions, and you want it to make sure it follows that set of instructions.
And then it goes for another round of it of of training, which is called preference tuning, and that's where you look at lots of different answers and you choose which ones are best. And they constantly do that. So if any of you guys have been using ChatGPT for a little while, you noticed that within the last, like, month or two, there was another update to the preference tuning, and it's changed its responses quite a bit.
I think it's actually quite a lot better than it used to be. But, yes, we do this thing called preference tuning, and that's how we get here. But we're still only predicting one word at a time. That's great, and that can get us to a point where we think that these LLMs are very clever, but it can't do anything.
Can't do anything. It just, like, gives you words. This is where agents come in. So often we talk about agents, and sometimes they're used interchangeably, but they should really be quite different. Agents and LLMs are slightly different. Agents are LLMs that have some reasoning capability and can make consistent plans.
So, I you ask it to do something, and it will have a think about it. Oh, I should do x y zed, and then, actually, okay. So, this is the order in which I should do it, and then you give it the ability to access tools.
Right? And this is where a lot of the MCP stuff that keep people keep talking about, model context protocol, comes in. It's basically, how do I allow an LLM when it knows it should execute something? How do I allow it to be able to use that tool?
How do I allow it to search the web? Or how do I allow it to use a calculator? And this is where we can now count on LLMs. If I ask an LLM to do some maths one number times another, I wouldn't trust the output.
But if I know that it's got a tool attached to it, like a calculator, then I'm much more confident in using it. And that's what agents do. So agents are LLMs that have some reasoning capability, and they have access to tools that allow them to do things in your work, which is why we're even more able to do stuff at work with them that allow them to do things that are useful.
So there we go. That's agents, and that was, like, end of twenty twenty two. We started getting those agents, and I know there's a talk tomorrow about the state of agents in in twenty twenty five from someone from OpenAI. So we'll get more information and context on that.
But that's largely it. That's how we got to where we are now. Right? So that was pre two thousand and ten, and then we started to see those milestones getting to, like, twenty thirteen. Understanding language. Right? Understanding what words mean things together, and actually processing things in parallel.
So that's how we get to then LLMs and agents are just LLMs that have some reasoning capabilities and tools. It's cool stuff. Right? It's cool. I mean, I like it. I like it. I don't if anyone else does, but I love it. I love it.
But that's part one. Like, I said, I didn't just wanna indulge in, like, academic stuff. I I just think that's cool. But I wanna give some, like, practical tips for people that may not use LLMs much, may not use prompting, and also might wanna use them in in applications.
How how should we put these together? So, first off, some prompting tips. Right? Be explicit. Be very explicit. Right? If you ask me to write a story, right, that's a bad prompt, you ask me to write a story, you don't know what's coming.
You don't know what's coming. I'll write anything. I can talk a lot. It'll be a it'll be a whole load of waffle. But, if you ask me to, let's say, write a hundred word story that's funny about cats stealing pizza, and include a surprise ending, it's it's better.
You could even go further. You can say, hey, look, I want scene one to do this. I want scene two to do that. Like but what the the the main thing here is be explicit to the LLM. The more explicit you are, the better the output you're gonna get.
Alright? I do a lot of coding, and when I'm not explicit about what I want it to output, it will just give me a whole load of rubbish. Right? All those people just say, oh, yeah, I did it in five minutes, and I've got like this big web app and stuff.
Yeah. Yeah. Yeah. Yeah. Yeah. But what was your prompt? What was your prompt? So you need to be explicit about it, because otherwise, you'll be iterating, and you'll be in a whole world of pain if you're not explicit. The other one is define your needs.
This is something that, like, I guess, is maybe less obvious, but, like, you can ask the LLM to specify the format or the tone or the style. Now imagine, for example, you're building, a customer service chatbot. You know, you don't want it to be condescending.
You don't want it to be there like, what do you mean? You don't know your four hundred and eighty six digit customer reference number? You don't? Fool. No. You don't want that. So, actually, when you when you prompt an LLM, one of the first things you do is stick in what we call the persona.
So if you look at, like, a cursors one, which is a a development environment for coding, They have something like you're an expert coder, and you are giving help on the world's best IDE program. That's the first thing they have in in their prompt.
And that's what we do. Right? I when I prompt at Spotify, you are a music expert. You should do x y zed. And I also want you to speak in a friendly tone. I want you to speak in this, or I want you to be very concise.
So be explicit about the persona that you want the agent to talk to you in. Alright? That's a really important thing, and and especially if you're gonna get this out to in front customers, in front of other other people, make sure you specify how you want it to speak.
And that's how you get, like, the output that you want. And I've put here quality in, quality out, and, you know, it's it's a more optimistic version of garbage in garbage out, you know. I wanna be optimistic here, you know. We're positive people.
Positive. Right. What about if I wanna build it for an application? I wanna build it and I'm I'm I'm writing code. Forget about the code for now, if you ever have a code for now. First thing is specify the output format. Alright? Incredibly important.
Like, if your if your application is only expecting, like, one word, let's say that I'm writing something to classify, let's say, it's still a customer service bot. Let's say it's still the customer service bot, and I wanna change my tone depending on how the customer is responding to me.
Because I might get angry as a customer when I'm not getting the thing I want. So I wanna find out if I've got a negative sentiment or a positive sentiment, and I'm building an application to respond to that. Maybe all I want is a single output.
Negative, neutral, positive. Tell it. Tell the LLM you only want a single word. Tell it you only want these labels. Right? The other thing that's really useful is JSON. So if you've written a bunch of applications that need to speak to other services and stuff, JSON is a common format in which we transfer information, and many of the the tools so here's a code snippet for for in Python from the OpenAI one where I'm using chat gbt sorry, gbt three point five.
The second argument is a response format. I've just said, I want a JSON object. So make sure you tell it that you want the output, because then you can guarantee what you're gonna get out, and you can guarantee that your downstream service is gonna be able to read what comes out the LLM much better.
And there are different frameworks that support this. But, yes, when you're dealing with LLMs, you're not always you don't always want just a paragraph of text. You might want it in a specific format, and that's the way to do that. Next one. Prompting strategies by learning by example.
You might hear things such as in context learning or zero shot examples, one shot, few shot. You may hear hear these things. Basically, there's like it makes it sound complicated. All you have to do is give the LLM some examples. So zero shot is when I don't give any examples in my prompt.
Right? So let's say that I wanna find out what is the capital of France. You might just type it in. Often, you'll just type that in, and you expect the word Paris. But I guarantee, if you stick that question in to any of your LLM applications now, you might get something like rather than just the word Paris, you might get the capital of France is Paris.
Paris is this latitude, longitude, blah blah blah blah blah. You might not want all that. And I'll go on to a few slides as to why you won't want all that. So what I can do is a one shot, a one shot prompt.
This is where, within the prompt, I give it an example with an example answer. So I will start off and I say, what's the capital of Scotland? Edinburgh. That's what I want. Now give it the question, what's the rest of the prompt? What's the capital of France?
And hopefully, this time, it should just output Paris. And so if you know that you want a certain type of output, give the examples in the prompt. It's called in context learning, and it's huge. I do it whenever I know there are gonna be edge cases that I know that I want the LLM to make sure it outputs the right thing.
Give those edge cases. Give the examples that you need, because then the LLM will do much better job at providing the output that you need. And then few shot is just the same, but just apply more examples. Just just do more examples. Right? Not just one.
Okay. Cool. Cool. Into context learning. I use that a lot. I use that a lot. Right. Okay. So up until now up until now, I've spoken to you about making the prompt a bit longer. I've said, be a bit more explicit. Right? Define your needs. Define a persona.
Give examples. But as you can probably imagine, the longer the prompt, the more computational power and you get less efficiency. So the processing time increases, and up until about a thousand input tokens so I said that transformers process input tokens in parallel. Right? They process input tokens in parallel.
So up until about a thousand input tokens, you'll notice that the output comes out at the same amount of time. Really, it doesn't change. Beyond that, it takes a bit longer. Bunch of stuff happens on the on the on the back end, which I don't fully understand, but you can see a lot of benchmarks.
When we tried it also at Spotify, we found we started going past this limit, and it starts to take a bit longer. So speed yeah. The speed, it gets a bit slower beyond a thousand tokens. But importantly, output generation. Now I've said just if you want one token, have one token.
This is because of the way that an LLM outputs information. So let's say I have a prompt. Takes a prompt, everything goes in parallel, and then it just generates one token. Remember, language models just predict one well, I say token. I'll get to token on the next slide. One word.
Let's use word for now. You just output one word. So how does it generate a full paragraph? Well, it takes that one word, puts it back onto the end of your prompt, puts it right back through, generates the next word. And then it puts it back on the end, generates it, and that.
So the more output tokens that you generate, the slower your application will be. That's a given. So if you want a quick application, reduce the number of output tokens. Right? That is a huge or I mean, reduce your input tokens as well, but the output tokens are much more are are gonna be a much bigger factor in how slow your application is gonna be using an LLM.
So a bit like Albert Einstein says. Right? I'm gonna say, like, make it as concise as possible, your prompt. Make your prompt as concise as possible, but just not any more concise. Right? Make sure you if you need that example, that in context example to get the output the LLM to output what you need, put it in there.
Right? Make sure it is there, but don't put in five examples if you only need one. Right? So, this whole balance of, like, getting the prompt the right length is really, really important. So, yeah, got got time. Got time. Tokenizations. I've I've at the moment, I've just said I generate a language model of words.
That's not actually what happens when you talk about tokens. You'll hear that. And that's because when a language model looks at text, looks at the prompt that you give, not all words are split equally in one. So the the text here is my fitness tracker said nice job after I go out to get snacks, blah blah blah.
You'll notice, for example, that when I tokenize this, this is the colored lot, tracker is split into two. So tracker is two tokens. Right? So when you start looking at costs, when you go on the pricing information on your on those these websites like OpenAI and stuff, they will charge per token.
So tracker is not one token, it's two. And so you can look at some different visualizations. I've left the token visualizer on the link there, and you can get these slides from me if you want. Like, I'm more than happy to send them out.
But, yeah, you can get a visualizer to find out how different LLMs tokenize different prompts. But, yeah, one rule of thumb is one token is about naught point seven five words. And it's important when you're thinking about, okay, a lot of LLMs have a context length of, like, a hundred thousand tokens, they will say.
And I was always, oh, what about if I do it too long? And then I think about my thesis, and that was, like, fifty thousand words or something like that. An LLM could fit that inside a prompt and have extra room. So, there's a lot of room you can add to a prompt, like, could stick in a lot.
But make sure you know roughly how many tokens you're using. But often, I like to think in words, and then I break it down afterwards. And this is just a graph showing some more examples of, like, how the response time compares depending on the input length.
So the orange bar is a ten is a hundred input tokens. The blue bar is a thousand input tokens. You can see that they're roughly the same height, meaning that the difference between a hundred and a thousand tokens doesn't change much. But as soon as you get to ten thousand input tokens, the latency, this the it takes a lot longer.
So, this is just an example showing that there is a relationship between latency and in and the number of input tokens that you have. Costs are the same thing. Right? Like, essentially, most companies will do a per token model, and then output tokens cost more.
Remember I said that output tokens are generated in sequence, which means the amount of computation required to produce output is much more than your input. So I went on OpenAI pricing thing last week, and down the left hand side is the input to cost per one million tokens.
So if you just look at the first row, for example, it costs two dollars per million input tokens, but it's eight dollars per million output tokens. This is because of the increase in computation that's required. So remember that. Having more output tokens makes your application slower and more expensive.
Don't produce more output tokens than you need. Right? Make your inputs as as small as possible. So, yeah, hopefully, that's like fairly good tips, you know. I I do have more, I just didn't have a lot of time. I even have more like one minute left, like I this is crazy.
I have I have more. Come see me if you want more. I love this stuff. I really do. But, yeah, we went over some like prompt engineering techniques. So you looked at, like, learning approaches, like zero shot and stuff and performance considerations in terms of latency and cost.
But I wanna just quickly go through, like, how I use LLMs, and I'll breeze through this because I I I have zero time, essentially. But, yeah, creating slides. The whole thing I did here, like, I brainstormed the Gemini. Was like, what's the title of my talk?
What should I talk about? Come, let let's do this. So I I I brain this is a screenshot of me, like, making brainstorming this with with Gemini. And then I just took that. I took that what we got as output. So it said, hey, look, look, section one will be this title slide.
I just stuck that into Gamma app. It just created the slides with a prompt. Look, these words make no sense. Look at words on the side. There's no sense. Why? I use the cheapest LLM to generate these images just to make it look like I was doing something.
Like, yeah. Like, I use LLM to do slides, man. It's productivity. I wanna be that 10x guy. Oh, I'm out of time. I use it a lot to do coding. Right? This is why I got Look My No Hands. I can code without typing anything.
I just put in a prompt, press go, go to my meeting. That's how I roll. That's how I roll. And then also, sometimes when me and my daughter don't want to do books for bedtime, we make up prompts for stories ourselves. Like, I say, what characters do you want?
So this is one that I did with my daughter as a screenshot of, like, we want a robot named Rusty and a dragon called fiery Freddy and a unicorn and all these things. Like, we can do bedtime stories together. I also thought, what do I look like as a Muppet?
You know? And and, like, I don't have hairy shoulders. Like, come on. OpenAI, they have mugged me off. Mugged me right off. Yeah. Yeah. See? But then I also look. I I during COVID, I wanted to grow my hair. I started balding, and I was like, okay.
That's not gonna happen. Started shaving my hair off, but what would I look like with hair? You know? I can imagine a world where I was a different person, where my hair grew, you know? That's this is stuff that I like to do and think about.
And I am way over time, so I'm a leave it there. Thanks for listening. Appreciate it.
