Single cell RNA sequencing is a paradigm-shifting technology that has changed how we make discoveries in cancer, cardiovascular and neurodegenerative biology. Previously only biologists equipped with computational and bioinformatics skills could use it. At Biomage, we have changed that.
This is a story about a small team of mission-oriented engineers, bioinformaticians and researchers who have reduced single cell data analysis time from several months to a week and made it available to every biologist to do entirely themselves. How did we do it, what did we learn, and how can these lessons be applied in other engineering contexts?
Auto-generated transcript - may contain errors.
Tap a timestamp to jump the video.
Thank you very much, thanks everyone. I'm happy to be here, really excited to tell you more about this. So I'm Eva, my background is software engineering. I love solving problems, especially those that make a difference in people's lives. And this is why I created BioMage.
Today I'm going to tell you a story about a really big problem currently in high throughput biology and how this problem is really impeding scientific research. I'm going to tell you how I and my small team at BioMAGE built the first open source platform that helps with this problem and completely removes the bottleneck of analysis that researchers were experiencing.
I'm going to tell you a bit more about the technical challenges that we faced while we were working with this, how we made it scale, and what's the impact that we saw and what's next for us. So, before I do that, let me take you a little bit and briefly tell you some of the history that's been happening in the area of high throughput biology for the past twenty years.
So, in April two thousand and three, an international team of scientists all over the world completed a thirteen year long project which was aiming to sequence and map all the genes from a couple of members of our species, and they essentially built, for the first time, an entire genome blueprint for building a human being.
So this essentially enabled the field of precision medicine, and now what we can do is we can take a certain a certain type of cancer, and we can build a specific treatment that's tailored to the specific person and is going to work better for them.
And that really improved the results in terms of cancer treatment, and it unlocked an entire new field in the area of precision medicine. But that didn't stop there. So in twenty eleven, bulk RNA sequencing went mainstream. What it allowed us to do is to see the expression of genes in really large cell populations, That essentially started off the field of discoveries of biomarkers.
So what that means is that now we can look in a population of cells, can look at what genes characterize them, and we can predict whether a certain cancer treatment or a certain treatment of disease is going to work or not. And as you can imagine, that revolutionized cancer treatment and basically drug target discovery in general.
Now, very recently, just three years ago, single cell RNA sequencing went mainstream, and that's a new type of RNA sequencing that's very much like bulk, but with one little difference that makes it basically game changing. And that's the fact that we can see the gene expression of many genes for every individual cell.
And what that basically did is that it revolutionized the area of rare disease discovery. So, for example, one of the recent things that our collaborators of our scientific co founder at Harvard Medical School found out is that they found out a new cell type, a new rare cell type in human airways that nobody knew about.
So what they did was they did a single cell sequencing experiment taking and studying the disease of cystic fibrosis that affects about seventy thousand people all over the world. And they found out that there is this rare cell population, it's called pulmonary ionocytes, and it expresses this gene.
Then they went back, they did an experiment with mice, and they found out that if you switch off that gene in mice, they experience similar cystic fibrosis like symptoms. So that essentially gave the world and us a new treatment for cystic fibrosis. That's just one example.
Now, single cell sequencing has a huge potential to revolutionize cancer research, but there are still a couple of things we need to do to get there. And one of them is that we need to resolve the big bottleneck which is in data analysis.
So, let me tell you a bit more about this and why it is a problem by telling you a bit about Vicky's story, and Vicky is somewhere in this audience. Yeah, hey Vicky. So, I met Vicky at Glasgow University. She was a research group leader and she's a biologist, so she specializes in immune cells, immunology, and she was studying a particular type of immune cells that can be used for cancer immunotherapy.
So what Vicky did was she decided to do a single cell sequencing experiment, and when I met her, she looked really confused and really lost because she just did her experiment and she realized that the data analysis of that experiment is going to take her much more time than the rest of the experiment.
So, what Vicky basically did is she got the tissue from human lungs, then she went back to the lab, then she used a special solution where she submerged her cells in from the tissue, that solution basically separates out the tissue into individual cells so that later on the single cell sequencing experiment can be done.
So, Vicki did that, then Vicki went to her single cell core facility at Glasgow University, she handed in her sample, and she asked them to perform a single cell sequencing experiment for her. A few days later, they came back. They handed her back lots of files, lots of data, lots of genes, cells.
So that was about ten thousand cells and about twenty thousand genes, and for each cell and for each gene there was some information, what's the level of that gene, how many cells and so on. So that was about twenty million data points. And Vicky, who is not a computer scientist, she's not a biologist, had to somehow take this data, look at it and analyze her experiment and find something meaningful.
So how was Vicky supposed to do that? How is anyone who is not a computer scientist supposed to do that? So Vicky had two options. She could go and she could try to teach herself R and some programming, but by speaking with other biologists who did that, we know that this is going to take about nine months of effort, really hard work, full time, and Vicky cannot afford to do that.
She's running a whole lab, She has other people depending on her, and she's a biologist. Her drive is biology she doesn't want to be a programmer. So that really wasn't her option. And now the second one, and the one that she went with, was to work with a bioinformatician.
And here is what she did, she found a bioinformatician at Glasgow University, a really helpful one. I actually know him, we meet every now and then, he's very helpful. And she worked with him to do this analysis. So, what she did was she had to explain the entire biology of her experiment, what she was trying to do and why to him, so that he knows what's going on.
And then this was followed by a really painful series of emails, toing and froing, computing plots, computing analysis tasks, going back to the basics, and overall this was a huge pain. It took Vicky about six months to do all of that, to analyze her data.
And she was really lucky actually, because she did have a very capable and very strong bioinformatician to help her, but there are many biologists out there that are not lucky like Vicky, and they have to wait for years to get their data analysis done.
So Vicky thought, just like us, that this is really unacceptable. We cannot afford this. This is stopping us from progressing. This is stopping research. This is stopping treatments. And last year she quit her job as research and now she's part of our team.
She's helping us build that software. She's helping us improve data analysis and make it faster and available to everyone. So here is what we did. We went building and we built the first open source cloud based web platform for single cell sequencing analysis where a user without any bioinformatics or programming skills can just come in, log in, upload their dataset, click analyze, and their experiment is there.
It's processed and it's analyzed. So, without that platform, if Vicky was to do this herself or any other biologist, what they would need to do is they would first need to take in all the files from their core facility after the experiment was sequenced, and then they'll have to combine them into one specialized R object.
Then what Vicky and other biologists will need to do is they'll need to write a custom data processing pipeline that works with big data, big files in R and for that pipeline they would need to write filters that go over that R object and remove all cells that are invalid.
So, all cells that were introduced by error or all cells that are actually dead cells as a result from the experiment, they would have to be removed. And after each filter, Vicky would have to compute a plot that shows the results, what was filtered, why it was filtered, some statistics.
She would have to do that for each filter. And after all of that was done, then this comes the main part, the data exploration, where she's finally ready to actually look at this data set and try to find out what's happening. So, she would have to develop an interactive way in which she could see different visualizations of her R object and go over them together at the same time and find insights.
And now, the dataset we're talking about, after all the filtering is done, is about one gigabyte, but this is just for an average project. We've seen bigger ones, we've seen ones that aren't too big to be even able to be on the machine of a biologist or of anyone with a regular machine.
So, this basically, at some point, becomes intractable. With our platform, we completely removed this from the user, and now they don't need to worry about this. And for me, as an engineer, this is really exciting and really fascinating why I was developing it because each of those modules actually has very different requirements in terms of size, in terms of interactivity.
And I really want to focus now on data exploration because this one is very exciting because here we not only have to work with large data files, but we also have to make sure that it's all really highly interactive. So, let me tell you a bit more about this data exploration.
So, what you can see here is a different visualization of this R object. So, on the top left, you can see the embedding, and the embedding is basically this really nice picture of lots of cells, and under the hood we run some specialized machine learning algorithms that determine how close each cell should be.
So, what you're looking at here is a dimensional reduction from twenty thousand genes up to a two dimensional space where you can have cells, we take cells and we compute how similar they are based on the level of expression of genes they have and the set of genes they have common that are expressed, and then if they're really close on that, we put them close together.
And here is how we get the embedding. And this embedding typically on average has eight thousand cells, so that's eight thousand points on that plot. Similarly, we also have all the genes on the right side in the gene list too, about thirteen thousand of them.
The user needs to be able to go through pages, scroll, search for genes, select them. If they select a gene, they change the embedding. And we also have this heat map, which is a matrix of all the cells in the experiment, and for each cell, the expression for a particular gene.
So on average, we've seen that users use about fifty genes, only for the heat map that's about forty thousand points. Another way it works is that the user goes over the embedding, they check out some cells, they go over the heat map, that changes.
And it's all really highly interactive, and we need to make sure that all of this works immediately, there is no slowdown, there is no waiting time. And we did that, and we're the first to do that because all the previous solutions basically failed on the technical level.
So, let me tell you how we did that. So, all of this that we have here is powered under the hood of this black box that we call a worker. So, what a worker is, it is essentially some bioinformatics code written in R, which are all the methods, all the single cell sequencing methods, all the tasks, and they're written by our bioinformaticians in the team.
And this bioinformatics code works on this R object, which we also call count matrix file, and this is essentially a big matrix containing cells as columns and genes as rows, and it contains information for each of them. This matrix changes every time we run data processing, just because when we go back and when we change filter settings, we filter less cells and then this matrix basically changes and we have to go back and re download it.
And this matrix, as I said, is about one gigabyte size. All of that to make sure that it actually works and it communicates with external services, we've wrapped it around the Python wrapper. So, that's just some Python code that handles all the communication with S3, with our UI and everything like that.
We've put all of that in Kubernetes pod. And now what's happening is we have our S3 bucket, we have our UI, and every time the user wants something on the UI, say they want a heat map, what we do is the UI asks for the heat map and then the Python wrapper receives the request and it tells the bioinformatics code to compute that heat map.
Bioinformatics code uses that count matrix, are objects, does some statistical models, some analysis, and after a while it comes up with this answer and the answer is about forty thousand data points. It puts out that answer in S3, tells the UI that the answer is there, it's in S3, and then the UI downloads it and displays it to the user.
And the heat map appears. So, that's happening with the embedding, that's happening with the chain list, and now we placed all of that in, we're running in AWS, so we put all that in an EC2 instance, and because the count matrix file is different for every experiment, we also have one worker handling one user.
So, if you have two users, you have two workers if you have five users, you have five workers basically one to one mapping between worker and user. We put sufficient capacity, or we thought that it is sufficient, we thought that maybe fifteen workers running at the same time would be okay, and we made that EC2 instance run all the time.
It was an MVP, so we just did that. We wanted to place it out there to people and let them analyze their projects. We also wanted to be a little bit smarter about it so that we don't have to go and ask the worker for tasks all the time.
So, added an in browser cache to store some results and we also added a checker that first checks S3 for those tasks before it goes back to the worker. And in that way we eliminated creating workers when the results are actually already in S3 or in the browser.
We also had to do a little bit of cleanup, so we deployed this cleaner in Kubernetes that goes and automatically cleans up any Kubernetes pods that haven't been used for a while, so I think it was like one hour. So after one hour, the cleaner just removes that pod and clears up some of the capacity so that other people can come in and do their analysis.
And on top of that, we put this API to handle all the requests, all the communication between the UI and the worker, and we also made the API capable of starting those Kubernetes pods. And that was really cool. We released it. We tried to release it as fast as we can so that we get some feedback, and we noticed that it started making a huge difference.
One of the users that we got was Sini from Harvard Medical School, and twenty minutes in her analysis, she managed to find out that her cancer treatment worked. So what Sine did was she had two samples. One sample was all the patients with her treated cancer, and the other sample was all the patients before the treatment.
So, what she did is she did processing the platform, and then afterwards she looked at the embedding, she looked at the heat map, she managed to identify which are the cancer cells, so which cluster has those cancer cells, by using the heat map and evaluating which is the marker gene for that.
And then afterwards she went back, she looked at her other sample, which was with all treated patients, and what she could essentially see is that her cancer treatment worked. So, like this entire cluster with the cancer cells just wasn't there. Her treatment worked.
I as an engineer, the first time I saw this I was like, oh, maybe the platform is broken. But then we went back and we looked at it and indeed it was there, was working. You could go to data processing, you could download the statistics, you could actually see what's happening under her hood.
And that was really exciting and that brought us more users. Users started coming in from referrals, we got another user who was running a showcase of the platform to others, and she was trying to get everybody from her lab to use our platform just because it's so easy.
And we started getting more users and our EC2 instance started getting filled up and at some point, well, there was no space. So, we had to do something about it. We were thinking how we were removing the bottleneck of data analysis, but suddenly we started becoming another bottleneck.
We needed to empower biologists to analyse their data and we went back and we started thinking of options. The first option, which you probably can come up with as well, to just increase the instance size. So, we could just calculate how many, again go back, think about how many workers we want at the same time and just deploy this really huge instance that can hold all of them.
That's really easy, that's two minutes of cloud formation change, but it's not the best one because what happens when users go home and they don't need to analyze, then we have this huge instance that we have to pay lots of money for. And we are a start up, we started two years ago, we're fairly new and we're kind of short of money, I mean, we don't have money to spend on things like this too much, so we thought that maybe there is a smarter way to do this.
And the second option that we thought about was to try and run each worker in a separate instance. And it actually does make a lot of sense now, we don't have the problem of having too big an instance that's underutilized. We also don't have to worry about running out of capacity because as a user comes in, can just start the instance on demand.
But that was in a violation with one of our biggest requirements, which was that we wanted this platform to be interactive. We wanted when the user comes in to data exploration to start up this heat map loading immediately and not have them to wait for minutes and minutes for this heat map to load.
On average, starting up an EC2 instance, assuming all the packages are there, you've pre baked your AMIs, you've basically done all the work that you can, you're not using some super agent Windows or Unix machine, it's going to take about two to three minutes.
So that's two to three minutes, the user is staying there in front of the browser, waiting for the heat map to load in the best case. And we couldn't afford that, this was going to remove some of our best users, so that wasn't an option for us.
And now the next intuitive option was to add an autoscaler, so the other capability that allows us to have a couple of instances started up and ready for when the user comes in. But I actually did that at Skyscanner. We had this shared cluster that we were managing ourselves, our team, we were a team of six people, were the deployment team, and were in this shared cluster, we were running all the services in production, which was about five hundred services back then.
We had that Lambda function that was scaling up and down. It was running every five minutes and it was scaling up and down instances, so when we would see there is lots of underutilization, it would move microservices to other places and then it would just switch off instances or it would start new ones.
And I knew that it was a lot of work, was actually about four hundred lines of Lambda function code, it wasn't great, it was a lot of maintenance and I didn't really want to put my small team excited about improving human health to be managing an autoscaler, So, we didn't do that.
And what we ended up doing was outsource all of that, make sure that somebody else is worrying about it, and we chose AWS Fargate, which is essentially this layer on top of ECS and on top of EC2 that handles all the startup of instances and switching them down and worrying about how much capacity you have extra and so on.
So, all we need to do is we need to say that our instances are going to be managed by AWS Fargate, and then how many workers we want, how many spare workers we want. So, we configured, I think we have right now about four workers spare that we run, and as soon as we put that number, then Fargate takes care of the rest of it.
So we make sure that we start those extra workers. Well, Kubernetes does that for us automatically, and then Fargate provides the space for them to run. And now, maybe one of the questions you could think about is, and one of the first things that we had to resolve was, how do you know in advance what type of workers you need?
Because each worker needs to have a different count matrix file and this count matrix file really depends on the experiment. It's different for each experiment. So, how do you start a worker before you even know which user is going to come in and use that experiment?
And the answer is that we don't. We just came up with another type of worker, which is we call unassigned, and that's a worker that doesn't have that count matrix downloaded at all yet. And it just sits there in a pool of other unassigned workers without the count matrix downloaded.
And then what we did was we added capability of the API When a new user comes in and they don't have a worker, the API goes and says, okay, I'm going to take you to Kubernetes pod and you're going to be the worker for that experiment.
It takes it and it makes it download that matrix and turn from an unassigned worker to an assigned one. And then afterwards, we just proceed with the analysis as it was before and we get the heat map. What that essentially did is it unlocked this limit.
Right now, we can have an unlimited number of pods, unlimited number of workers. And two weeks ago, we had the showcasing of the platform in Edinburgh University, we had sixty seven users, and sixty seven users were running this platform together at the same time, which meant sixty seven Kubernetes pods that just started up, all the matrices got downloaded specifically to the experiment and everything worked, and it was amazing to see it, you know, after all this work, it just worked and they were happy and most of them also
turned it on engaged users, they uploaded their datasets afterwards, and now we have plans for even more workshops. We got so many user accounts as well, So, we released the MVP twelve weeks ago, we released the scaling about a month and a half ago, and now we have more than was from last week we had almost three hundred users accounts.
This week we actually have more than that. This metric is changing all the time. The number of engaged users is going up as we speak, and I was shocked to find out that one of our users from University of California, she performed more actions on the platform than our admin accounts that we used to test and develop that platform.
So, that was really cool for me. And, now it brings us to think about what are the next challenges? What can we do next? And we know that the field is moving really fast. There are new methods developing as we speak, and in order to succeed in our mission to improve human health, we have to allow researchers to use freely and easily those methods.
So how do we make sure that happens? Well, we can just leverage the fact that we are open source, we can make sure that we build a community where bioinformaticians can contribute methods, biologists can use them quickly, and then together, I'm pretty sure we can help treat cancer, and it's only a matter of time.
