Malted AI is on a mission to shrink AI models to solve the hardest business problems. Iain will talk through the challenges and lessons of building an AI startup in Scotland and raising £7m from Venture Capital. He will also discuss deploying Small Language Models (SLMs) in an enterprise environment and how knowledge distillation can help build more focused AI solutions at scale.
← All talks 
Build Turing Fest 2024
Distilling Small Language Models for Enterprise
Auto-generated transcript - may contain errors. Tap a timestamp to jump the video.
So thanks everyone for joining us today. My name is, yeah, Ian Mackey, and I'm here to talk about distillation. Sadly, not off the the whiskey variety, but hopefully something almost as cool, which is small language models. A little bit about my background, I used to be in kind of quant trading down in London, moved back up to Edinburgh, did my masters in AI at Edinburgh Uni, worked a bit in research, and then, you know, ran a ecommerce startup, I did my PhD at Glasgow on the intersection of search systems
and large language models, and then I'll tell you the rest very shortly. So in terms of the agenda today, I'm really gonna take you through a history of MultiDii. Firstly, the kind of genesis, which was winning the Amazon Alexa Prize onto founding the startup and kind of the first, you know, ten months of our journey, and then really go into some details of a kind of case study of us using distillation in practice.
So what is the Amazon Alexa Prize? So, basically, it's this big global AI competition. It happens every year. Around a hundred and fifty teams take par. And in essence, you need to build an AI agent to interact with real users on Amazon's US Alexa device.
So it's in a production setting, you know, has to be low latency, data secure, and you're dealing with real users trying to do real things that we'll go on to to shortly. And in essence, the users rate the system as well as domain experts.
And this was really where we developed our technology thesis, and we realized the problems that we as a group could solve. And more importantly than that, we realized what amazing things you can do with a small, awesome team focused on shipping great products.
If we look at the University of Glasgow team, you've got myself, my two now cofounders, Carlos Gemmell, CTO, Federico Rosetto, chief engineering, Paula Wucho, amazing machine learning engineer, Jeff Dalton, Alan Turingfellow, but also our chief scientific advisor, and then Sophie Fisher, who we're hoping to nick once she finishes her PhD.
So what is a task bot? So I think when a lot of people think about conversing with AI agents, people think about a kind of general chit chat system where you can have an open ended conversation about politics or movies of what you wanna do tonight.
And we'd kind of define that loosely as a kind of social bot. But what we were really interested in was actually working with the user, being a companion to help them solve real world problems that are complex and interesting. So you can imagine, in this case, something like there's a problem on a bike, and you as an AI agent actually assisting the user to solve this problem and reach a very specific goal.
So the domains that we focused on during Amazon Alexa challenge were primarily in the cooking domain as well as, like, you know, crafts and and and art as well as DIY. So like many people, before they kind of launch a product, you kind of sketch out what you think that your dream user experience would be.
So we started thinking about, imagine cooking a meal for four, and you've got Gordon Ramsey as your assistant, maybe minus some of the expletives. So he can talk you through, Oh, you've got these ingredients, you've got these preferences of people coming, maybe there's dietary requirements.
What is the dream menu? So you find your task, and then he walks you through it. Do you have scientific questions? Are you are you running into issues? Do you need to speed up or slow down? Really being that trusted companion grounded in factual data to help you through this interactive, fun experience.
We drew this out, and you can see roughly here it's about firstly working out what it is you wanna do, designing it, and then actually doing the task into the end. There's multiple phases of this, whether it's searching for the task, checking that it's right, walking through it.
But there's lots of, like, core components that you actually need to do this. This isn't is this is not trivial. And whether it's, you know, search, whether it's validation, whether it's walking through, there's there's lots of core components to a system to do these kind of complex behaviors.
And we kinda came to our first problem that was like, oh, crap. This system does not exist. So back in twenty twenty two, there was kinda two options, I would say. So one would be your more traditional kind of intent based conversational flow.
So if anyone's had had a really long and painful conversation with their bank, I apologize if there's any kind of people in banking here. But in essence, it's it's very brittle conversations where you can just tell someone is crafted if and if and if not, And this you know, it wasn't clearly fit for purpose.
So, you know, we drew out the requirements. This is not natural. This is not like having a human, you know, in the room helping you. It's really not flexible. If you get caught in the wrong space or or you're asked for something different, this this would fail.
There's some form of control, to be fair, like as a as a UX designer, you can craft these in some way. Obviously, we're deploying in a production setting, dealing with real users on US, Amazon, Alexa. Data security is massive, like we cannot send the data elsewhere.
You cannot send it to to OpenAI, and it needs to be low latency. So actually, those are two ticks here, but it's not scalable. Right? We're dealing with hundreds of thousands, millions of users. They can ask us to do anything. I I am not going through and making these if then else statements, so it's very brittle.
So back in twenty twenty two, another approach that obviously was emerging now, and now obviously there's clearly a lot of hype around, large language models. So something like GPT-three, so it's one hundred and seventy five billion parameters, very big API, and if you potentially did that with some prompting of task data, you can imagine this is natural.
These models are really good at generating fluent natural text. It's flexible to some sense. If you have different input, it will definitely generate your different output. But it's really hard to design complex experiences, never mind hallucinations, certain things are just very hard as a product designer to do in a very basic way.
It's clearly not low latency. If you're trying to especially chain multiple prompts together in these big models and you want, as a user, to get response back in two to three seconds, this is certainly not gonna happen. And unless you've got a spare three, four, five million dollars, you're probably not gonna be able to self host this.
And in a data secure environment like we were, we weren't able to share the data, so it wasn't even possible. But it is scalable. Because they're good at handling different contexts, if you are doing different tasks, it's scalable. So we effectively realized that if you look at the brittle end and the full flexible end, they almost had opposing pros and cons.
And we kind of thought, hey, can we do something that kind of combines the best of both worlds? So what our solution was was we created a network of small language models that, in essence, work collectively to to to get the types and rich responses that we needed for this flexible conversation flows.
And what is a small language model? So I would describe it as an open source model, something normally between one hundred million and, let's call it, seven billion parameters. In essence, you can self host on a small to medium sized GPU and go through the system at a high level roughly how this would work.
If you look at the offline pipeline here, we effectively would parse high quality web content to create these tasks. We would use SLMs in the loop to make sure they're rich and valid and vibrant. We'll go into that a bit later. We would then have functionalities, we call them.
But in essence, these are subcomponents of the system that are experts at just that one thing, and this could be searching for a task, ask you know, answering a specific category of questions onto something that maybe isn't a language model, which could be something like setting timers or interacting with lists, other functionality you could imagine you might need.
We had something, I won't go into too much detail, but kind of phase policy. So in essence, it's high level, almost like user design that we can fuzzily follow, which kind of gave us some flexibility relative to kind of intents. And really, the key here is we used a an SLM as the kind of brain for the system.
So we called this the neural decision parser. So this is the model that given what is happening in the conversation task, it can point you to the right subsystem to generate the answer you want. And now suddenly, if we kind of look through our criteria, we have a lot of the benefits of using language models without a lot of the kind of drawbacks.
So it can be highly natural, it can be very flexible. We're able to really control and craft amazing user experiences. It's low latency, you can self host, and it's scalable. We can handle tens of thousands, millions of tasks. So now to go into a little bit more of the actual task data.
So to power these types of conversations, we realized that just doing some kind of boring, linear step one, step two, step three isn't particularly engaging. So we needed a new data structure to represent the rich information that we need to ground our system on to do the user experiences we wanted.
So in essence, we created our directed graph, which we were able to model much more how humans would model tasks in their minds. So something like if you're trying to fix a bike, the system will take initiative and be like, oh, is it the front or back tire that's an issue when you go into different flow?
As well as things like mapping ingredients. So you can see here, milk, eggs, flour, and really grounding them into different steps. So in essence, you're giving the system a chance to know the science, what actually is happening in the real world so that you can correctly handle that that situation.
So we've actually crawled hundreds of thousands of rich task data to, in essence, power the input of the system. And let's go into a bit of an example here. So this is creamy zucchini pasta. And you can really see how rich and flexible we were able to make this task data, so you could handle complex situations and really take initiative, take system initiatives.
So something like here where you're saying, hey, how do you want your sauce? Do you want it firm or runny? And then we could handle that from a scientific, from a task, from a procedural level. We know what ingredients are at what steps.
So if the user doesn't have them, if they ask for substitutions, if they have dietary requirements, we at least have a chance of getting this kind of factually right from a system perspective. And we're able to store really interesting, detailed information that the system might need and we know is actually useful.
So it could be fun facts. It could be extra information. It could be specific jokes to that task. You can be a lot more creative. So we've got our we've kind of described it, the rich input. We've got our system here that's flexible.
But core thing that we need is, given what a user says, we actually know what it is they want we we need to do to to basically progress their task and benefit them. And what we did here was we used a small language model.
So this was a t five model, so about seven hundred million parameters, to, in essence, manage, be the brain of the system. So what it would do is it would take in the kind of conversational context, so what we've said, what they've said, particularly what they've said last, as well as that representation of where the task is, so, like, what the hell is actually going on in the real world or what we think is going on.
And we would basically semantically parse, we would generate function calls, this case it was in Python, to which of the subsystems we want to do, and then we would parameterize it based on what we thought was needed. So an example would be if the user said, hey, Shepherd's pie, please.
We think they're searching for a task. We're able to call the search system, and we're parameterizing it with Shepherd's pie. Or something like how much onion? We know they're on a step that they're using onion, and our question answering system can then reason that's a task specific question and go back and see, you know, two hundred grams.
If the user says, what's step four, it knows the user is wanting to know more details about step four, and we can return that to the user. So again, it's controlling that navigation so that those subsystems can be used rather than relying on these big general language models or other types of systems.
So this is great, right? So now we've got rich task data, we've got super flexible systems with subcomponents, and we've got a way of navigating it in an interesting and scalable way. But we don't actually have any data yet to train these subcomponents and the neural decision parser.
And anyone who's done anything in machine learning knows you need really high quality training data, so that's input output pairs, to train those subsystems for them actually to work. So this is really where we realized we did a different way of approaching this, and we used distillation as a way to create large quantities of high quality synthetic data.
So if we look at how, you know, traditionally this would happen in machine learning, you would get data annotators or or or domain experts to, in essence, go through the data you have and manual label it to get these input and output pairs so you could train your machine learning model.
And this is generally high quality, but it's definitely not scalable. It's super expensive, and it's actually very brittle. So if something changes your taxonomy, if the data changes, you need to suddenly get annotators in. You need to redo this whole process. Unlike our approach was actually, can we do it?
Now we've got these new large language models that have really interesting capabilities if used in the right way. Can we create teacher systems? In essence, these are normally a series of large language models working together to create these high quality training data, and these can be relatively unconstrained in terms of model size and time.
They can run offline. They can have models checking, filtering, synthesizing. You can do some fun and crazy stuff to make some awesome data. But the key here is then you take that high quality data and you train a much, much smaller student language model that then you can deploy on prem as scalable, as cheap, you can make it secure.
And it's really because of this distillation process that we can create high quality, scalable, cost effective, and actually flexible data. And it's all about moving these the task performance of the small language models up while keeping them super efficient, you know, super fast, and and kind of small models.
So let's kind of go through a bit of an example here because that was a little bit high level. So if we look back at the kind of neural decision parser, so again, this takes in where you are in the task, what's being said, and then it's it's basically creating a Python function call of where you need to go in the system.
So what we did was we realized we could not get enough humans to annotate this, and there's also issues with with actually privacy with private data to even do this. So we inputted all our raw task data, so this could be, you know, hundreds of thousands of tasks.
We did some initial seed annotations ourselves. We then needed to create the synthetic input. So we used large models, so in this case, a hundred billion parameters of this GPT three, to, in essence, create that input. So on distribution conversation that is on where it should be in the task.
Given this input, we then created other pipelines of of using these, you know, large language models to create the synthetic output, which in our case is the the Python function. And we used domain experts or or ourselves who became experts in the specific domains in the loop.
So we could check if there's issues, we could recalibrate the teacher system, and we could patch it so it was you know, got really, really high quality. And you can see this takes, you know, two to five minutes with some manual checking. It probably take a bit longer, but you get really high quality data.
And then what's awesome, we could then chain train our efficient SLM that is only seven hundred million parameters and can run-in half a second to the same performance as the teacher system. So now this is awesome. We have a scalable way of creating high quality data, and we have a super efficient small language model that we can deploy locally and is as performant as this big, much slower system.
Just to highlight the other benefit of now you're in synthetic data land, right? It's the fact that you can continuously improve the system. You have a way to create and augment data pipelines. So if the action space changes to our taxonomy of Python calls, you can make tweaks and rerun everything very easily.
If this was in the human land, it wouldn't happen. If new tasks, if new data's added, you can easily upload this, and and your system will adapt to that. And if you get user feedback and you guys things are going good and bad, you can loop this in either automatically or or or manually into the process.
So you're improving your training data really easily, really organically, so your system is improving over time. And, you know, it was really those kind of two core things that led us to to winning the the Amazon Alexa Prize, and that was the network of small specialist language models combined with a scalable way to create high quality data.
And you see here, these are, in essence, the user ratings. It's kind of continually trending upward, and we ended up around kind of four or five come the end, which considering the breadth and complexity of of things we're trying to do with real users, that was that was that was really good.
And then the the other thing here, you can also see as the conversations went out to kind of three, four, five minutes, we got better and better because we're able to handle these complexities and add value to the user. So, really, we after winning that competition, I think we'd learned a lot of technical lessons that we thought, hey.
We're onto something special now. We think this approach, this technology, the learnings, can really be applied elsewhere to help people solve, like, actual real problems. And secondly, we realized what you can do with such as, you know, a small, high functioning team kinda working together.
And, you know, we fundamentally didn't want to stop, and we decided to to form MultiDayEye. So a little bit of a kind of high level thesis is we think, you know, there's a ton of focus on big general models and AGI progress. But fundamentally, for a lot of tasks, these large language models are overly specific and and and really not the level of efficiency that a lot of people and particularly enterprises need to actually get value.
And what we want to do as a company is we wanna solve some of the hardest problems in enterprise. We wanna partner up with people that have the data, that potentially have the distribution, and we're able to help them distill their own networks of small language models to to actually get that that gap in performance that we're everyone's struggling with, you know, between a few people around around the conference, to reach that point that we can actually get value of AI.
And the way we think we can do this and what we're trying to build, you know, up in our our lab is being the world's best automatic distillation. Is is the fact that we can become super customized, specialized in people's data and people's usage, and really build these experiences much more automatically, much organically around people interacting with real life systems.
So a little bit of background. So we've been going for, yeah, ten months now. We're trying to build a leading AI company specifically in in Scotland, which I think resonates with a lot of people here. We think it's an amazing place to build an AI company.
It's some of the founding work on AI. We kept it alive when when no one else did. And there's amazing talent here that if we don't innovate, if we don't create really exciting, sexy jobs, they're gonna have to go to London and the Valley.
And I think it's kind of on us as a community to make sure there's an ecosystem to allow people to stay in Scotland, but also be at the top of the game and the types of problems they're tackling. So, yeah, a little bit of a ten month overview.
Yeah. We we've we're putting a lot of practice things we learned during exams and Alexa Prize. We've raised some initial seed capital from Hoxton and Creator Fund with participation from, yeah, a number of angels and industry experts. And we've actually taken these these concepts and this research and deployed it in regulated environments.
So in law and asset management, we've shown that our technology cannot be only be much more performance, so sixty percent plus relative to these general large language model APIs, but we can actually do it with much, much more efficient models, so twenty x more efficiency and allowing them to securely host these systems on their premise.
The next slide was really just to highlight, again, the types of talent and the people you can find in Scotland. It's amazing the number of conversations I have, particularly when you speak to Silicon Valley kind of VCs, the first question is, hey, when are you moving to Valley?
And I'm like, I don't plan to move to Valley. The types of people on commercial machine learning, if you look at people like Andrew Yates, wrote the book on Neural Search, Antares Antoine, world leader in meta learning, to people in software, people like Jim Croft, Richard Jones, multiple big exits.
We have amazing talent here. I think it's just about, you know, cultivating that. I'll give a little bit of shout out as well for anyone who is currently looking for a change in in career. I'd come and speak to myself or or Carlos or Laura afterwards.
We're relatively aggressively hiring and should be about twenty to twenty five over the the the coming months. So and I'm going to walk you through a kind of case study of our our kind of first product that we actually just launched in a couple of weeks back.
So really interested. If anyone has any feedback or thoughts, again, please catch me afterwards. So this is a complex question answering system, and here, we'll focus specifically on the medical domain. So if you imagine you're a medical researcher who has a complex question like, are patients taking angiostin converting enzyme inhibitors ACE at increased risk of COVID nineteen.
So imagine if you're doing research on a topic like that. You have unstructured data, so PDFs, PowerPoints, words, or just plain text, and you want to find that information. But you want more than that. You want it to be factually accurate. You want it to be returned to you in a way that you can consume.
So this is actually from from our end system here. You can see is we developed an answer type in a system that you can create these facets of interest, so no, there's no increase, potential effects, inconclusive, other guidelines, and you're able to classify and summarize the different documents that support these facets.
So this is this is actually really hard to do that we'll go on to later. So I guess a traditional approach for this type of problem over unstructured data would be kind of RAG. RAG's awesome, right? Improves accuracy, overall reduces hallucinations. I'll go through a kinda quick pipeline here.
So given your unstructured data, you vectorize it, so something like OpenAI embeddings, maybe you're hosting it on Pinecone. You then, given the question, look for the closest vectors as context, and then you use that as part of your prompting. So in this case, g p t four, again, it's really complex.
Actually, g p t four struggles with this answer type. But it's one point seven trillion parameters, and then you would generate the answer based on that context. And the kind of drawback of a system like this is it is super general. It is not domain specific.
It would likely miss a lot of that intricacies that you would need to find the information. It's unable to be consistent and traceable for that type of complex answer that requires maybe multistage reasoning, and you might want different language models to do it at that time.
It's, again, not efficient, particularly if you're trying to do it over multiple calls. You're not unless you're gonna be a really large bank, probably gonna have your own dedicated instance of this, so depending on what your data policies are, you may not be able to call it, and it would generally take a while given the kind of size of models involved.
So what our approach is is, in essence, we use small language models to basically distribute that kind of reasoning load so that we're able to find, analyze, and then generate very specific results given a question. So we look through our pipeline, and again, it's evolving over time, but we're able to expand it and enrich the query with other factual knowledge that we think will help find relevant sources.
We have multiple search systems that, in essence, are super trained to your data that are going to later. So whether that's vectors, learn sparse, term based systems, these find your initial pool of documents. You then have a re ranker that, again, is kind of super tuned to your system.
And then we have a much smaller generation head because we can do it over kind of multistage reasoning and and train it for what these exact purposes. We could do something with a distilled eight b LAMA. And through this type of system and creating these networks for the types of answers and processing that you need, we can make it super domain specific, each small language model is trained for the task and also on on on your data.
We can enforce consistency through verification and other types of methods. It's efficient, can deploy it on just a small number of GPUs, and we can enforce low latency due to the model size. But really, what the secret sauce is for these types of systems is how do you create that high quality data automatically?
And this is really for us, it's about being much more creative with large language models, generating, checking, filtering synthetic information, potentially with or without humans in the loop, so that you can go and create hundreds of thousands of of synthetic questions in this case, annotations to relevant documents.
So given that question, what doc what document should actually be relevant, and then the cited answer. And given this information, you know, you can really, really tune the system. So I'll go go through it a little bit, but we would, in essence, take the raw documents, which could be off the the customer in essence.
We would then use an ensemble of teacher systems. So these could be things like large MISTRAL models, large LAMA models, but other types of open source models that we can deploy on their prem. Here, we'd be looking to create a really large diverse set of questions, but ensuring it's on distribution, production, because otherwise that would then be bad data in essence.
So given these multi step pipelines, we have these set of questions, we then need to find the documents that our models think are actually relevant. So in this case, again, would be an ensemble of different models, it would be assessing that document A and document B are relevant, document C is not.
Again, we'd then be looking to create these complex answer types that would involve summaries and then bringing everything together. And now suddenly, you've got this really amazing, rich dataset that would be almost impossible to do with with kind of manual annotation. And you see here, the pipeline's, I don't know, eight minutes, and uses a number of relatively large models.
We can then train systems that, again, are in seconds, are small and super focused. So if we look at some of the initial results here, and this is in the re ranking setting, So you can see some of the baselines, things like OpenAI and Cohere, and generally they're strong because they've got proprietary general data.
You can see our teacher system is really good, but it does take a while. This wouldn't be suitable for production and would also probably be relatively expensive. But what we can do is we can start from an open source student system and then use distillation to greatly improve performance while this is only a few hundred million parameters model that can be run-in seconds with a very kind of small GPU.
So you can see here how suddenly, you know, enterprises can deploy smaller models that they own that are safe on their premises and get really high quality results. And obviously, again, as their data changes, as their use case changes, you can make updates relatively automatically on those distillation pipelines, and you should see both these systems getting better and better and better with data and usage.
So just to kind of recap and summarize, I would say that smaller can be better, and people should really explore this area. Scalable high quality data is absolutely essential for any applied machine learning problem. And lastly, maybe more most importantly, I think Scotland is super well placed to to really kind of lead AI innovation, and I think we should all kind of get together to make sure that we can, you know, take advantage of our our benefits in this space and at this time.
Thank you.