Uptime matters, but so do your people. At Intercom, keeping our product working well at all times is critical to the success of our business. This means having people on-call around the clock. However, as we grew, we realised that we had ended up with an on-call setup that we weren’t proud of. We decided to solve these problems by creating a virtual team that would take over out-of-hours on-call work, consisting of volunteers, not conscripts, from across the engineering organisation. This talk goes into the process we applied, the positive impact to our on-call, and lessons learned.
← All talks 
Build Turing Fest 2019
Volunteers, Not Conscripts: Fixing Out-of-Hours On-Call
Auto-generated transcript - may contain errors. Tap a timestamp to jump the video.
Thank you. Hey. Thank you all for coming. We're gonna be talking about out of hours on call. And, really, this is a bit of a story of what we did at Intercom to take the problem of on call and improve it and make it better.
It's a story of something we're like, work we have done for about the last two and a half years. But also, like, I think there's a lot of stuff in here that is it's can be an inspiration for others, that there's lessons that can be learned.
There's a lot of stuff that's applicable to organizations of all sizes. So I think it's a good story. I think it's something that can be applied elsewhere, and it really benefited the quality of life of the people Intercom. So the thing with on call is it's it's really great, and I love doing it.
And a lot of my career has been spent doing on call or being pretty close to on call or organizing teams to do on call. So just as like a a quick kind of user interaction thing, who here has done on call, like in the twenty four seven days or whatever?
Cool. Cool. Is is anyone on call right now? It'd be pretty cool to just do this talk of like somebody gets paid and we all help each other out. Probably a better talk, I don't know. But what I love about on call, it's like a perfect storm of ownership, of technical challenges, things tend to break in interesting ways and that's kind of fun, but you're also responsible for your tech stack and for the customer experience of what you've been building.
But also, it affects the experience of your engineers a lot as well. So it's a learning opportunity of where when something breaks or when you find out a new problem or something unexpected happens, that in itself is just this great opportunity to dive deep and understand things more and fix things and document things.
And of course, if your customers are getting impacted by the problems that are causing pages in the middle of the night, then you being on call is very satisfying or it's great opportunities to do something about it. So it connects you to the customer, and it's very satisfying to fix something that customers are experiencing which is bad.
You go in, you get paged, you roll back the problem deployment, whatever. And and so that in itself, like, helps the customer connection between you and what's going out and what your product is doing. But also having worked in a number of fast growing companies, I think OnCall is where you see a lot of early warning signs of kind of systematic problems in the scaling of systems or maybe the assumptions made when different things were being built have changed or that processes aren't scaling.
Another reason why I also love OnCall was, like, before I joined Intercom, I worked in Amazon and I ended up doing becoming a call leader, which is basically when any part of Amazon breaks, you are the organizer of the call. And so this for me was like this great growth opportunity.
It opened up, like, with different teams. It's allowing me to grow and build reputation. So it was very valuable for me, and so I've seen a lot of value both in Intercom and Amazon of being involved in organization and the day to day work of the on call process.
So that's been pretty good for me and it's pretty good for your organizations. But also at the same time, on call is absolutely terrible. It can really make your life considerably worse before before you've, started going on call, especially if the organisation is growing fast or not paying attention to the causes, of problems or the quality of the operations.
And being woken up in the middle of the night, it's never fun for anybody, and it causes emotional and physical and mental stress. I can still remember the ringtone of the phone that used to go off when something in Amazon would break. In fact, my wife also can remember the exact ringtone.
So this is like emotional scarring from being pestered because some computer broke somewhere. And so even now, when I do on call, I don't tend to sleep well. I'm always kind of second guessing myself in my dreams that something has gone off and I haven't noticed it or something.
That's despite having years and years of experience having done an on call. It's still this unease of once you know you're on call and once you know you have to respond to something that it just tends to get into your mind and not allow you to kind of relax and sleep and that's pretty important stuff.
I mentioned that incidents and outages are learning opportunities, but they can also disrupt significantly the work you were planning to do. So in the middle of the day, something breaks, you can completely disrupt the meetings that you're gonna do or the code you're gonna ship.
And when things when things break, it can disrupt your team's plans, it can disrupt your own commitments to your own teams. So unless you've built in a lot of Slack and, you know, you've done things to to allow for these kind of things to break.
It's just dissatisfying to be constantly firefighting. And so it it's it's can be organizationally difficult to commit to things when you've got an unstable environment where things can break all the time. You can also kind of get depressed because of on call. When you're suddenly very aware of all the technical problems and all the things can break, you kind of can get overwhelmed.
You wonder, does anything work all the time and how can we fix anything? So on call really can bring out the worst and the stress that it can create and some of the situations you kinda end up in can produce overall negative vibe of how you think about your job, your organization, and your technology stack.
So as revealed in the lovely introduction, I work for Intercom, and so we're a customer messaging startup. I joined about five years ago. We just had our eighth birthday based in Dublin, San Francisco, London, Chicago, a number of and a number of other places.
And we're growing we've been growing fast through my my time there. So we provide software as a service to our customers. I think here's our messenger. This is nice stuff to show up on a screen. There's a lot more back end and interesting components when you're on the other side of the messenger, but they don't really work as well in presentations.
We do have a stand downstairs at the area where all of the vendors are, so do go there and check out our stuff. While these fancy GIFs and cat emojis and all sorts of things work pretty well and they look well up there, There's a significant back end and significant infrastructure that needs to be run for all of this stuff to keep working and for you to talk to your customers.
So I'm gonna give a little bit of history about who did on call at Intercom over the time that I've been there basically, and how that kind of grew and sort of what problems we ended up having. So it grew organically is the kind of easy answer for the start.
So we have a culture of strong ownership in Intercom. We have strong teams. They own their stuff pretty strongly. This partly came out of a bunch of people at Intercom and a bunch of leaders being, say, ex Amazon who famously put a lot of emphasis on full stack ownership within individual teams.
So as Intercom continued to grow, we built more things, we built more teams, and they tended to own those things deeply. But and this kind of seemed to make sense at the time. We would build something new, the team would go on call for us, and happy days.
And so out of I was on call, we've just kind of grown off on an ad hoc basis organizationally. Some teams would have significant amounts of us, some things didn't. We had a team of an infrastructure team of systems engineers where they tended to congregate and were pretty good on call for the wider infrastructure.
But then we had silos of, say, our email delivery team or our user data storage team. So it all made kind of sense at the time. So when we were less than ten people, just our CTO was on call. When we were about twenty five people or so, there was the systems engineering team.
And then but as a lot of as we got more and more engineers, we ended up just more and more and more teams on call. And that's a lot of people on call. So we noticed some problems with this. There were too many people on call.
So I mentioned just back to my experience at Amazon again. I I reflected at some stage that we had more people on call at Intercom than there were on call for Amazon's s three service. So an absolutely huge service that's, like, completely outstanding scale.
Yet we were putting more people on call and putting more people at the other end of a pager just to run Intercom. And Intercom's pretty much all entirely in the cloud. We're well architected to survive problems. We're not dealing with individual hardware problems and things like that.
So this seemed mismatched for the size of the company, the scale of our operations, and even how we build things. So the fact was we had lots and lots of people on call who were all busy, like, ready and waiting to get paged, and that was dissatisfying.
There was also an inconsistent experience of being on call and the quality of the operations between teams. So different teams had different approaches to alarming, documentation, whether we're runbooks, onboarding into on call. And so these kind of differences between how teams did that became a barrier to organizational fluidity.
It was difficult to move engineers from one team to another without them having to pick up the stack or figure out how things are done in that team. And so Intercom, we want engineers to be able to switch teams without a lot of friction.
We want to be able to reorganize our teams, react to our changing business priorities. As I said, Intercom is fast growing, so things can change pretty quickly, we don't wanna get stuck with teams that are inflexible or engineers that find it difficult to move from one team to another.
So effectively, didn't want to have operational moats around teams. Another thing we observed was Teams seemed to have a curiously high tolerance for out of hours pages. And what this meant was during the weekend or in the evenings out of office, Teams would or people would receive a page and just go, oh, yeah, that's a system that's a bit flaky or whatever.
And it wouldn't necessarily go in and fix it. It's you know, it was owned by a small team, so they were all kind of used to things which kind of break and which things are important and which things are kind of less important but still paid you.
And so this didn't seem right. Like, we've we really wanted to people to go home at the weekend, do stuff which isn't work, and not be looking at these kind of low quality alarms coming in and having to do something about them or just kind of writing them off.
So some teams were pretty ruthless about how they dealt with their pages. Other teams, they're getting paid a lot but didn't necessarily do a lot of follow-up with them. And so we ended up with just a real mix of teams in terms of the amount of stuff that they're being paid for or the amount of pages they were getting.
So this kind of inconsistency wasn't great. And we figured that this was eating our engineers, like it was impacting our lives, productivity, their general well-being. Like I mentioned before, the team's productivity will sometimes be greatly damaged by stuff that would kind of break or whatever.
And so while it's a lot of fun building stuff in a scrappy startup fashion, as Intercom kind of grew and grew, we kind of wanted to take a more professional and more like a broader approach to this. So we've kind of transitioned out of the scrappy phase of startup life and moving towards building a sustainable, high quality engineering team that that we wanted people to build careers in.
So our on call setup was really at odds with our ambitions to build a great company. So we decided to try and fix it. And so there's a bunch of slides here, it's going to look very easy, it's just like five slides or whatever.
It was like months of work, was lots of meetings, lots of chats, lots of documents. But the gist of it is that we built a working group of a mix of engineering staff and engineering leaders, but mostly individual engineers, to kind of think about the problem deeply and figure out what are the ways that are consistent with our engineering values and how Intercom works and figure out what are the problems we need to solve now and other problems that we can solve down the line.
So we really started from scratch. We looked at why we had to have on call at all and, you know, can't we stop doing can we just stop doing it? Ultimately, we end up building a model which we think was gonna work for us and then figure out a rollout plan to to roll it out across the organization.
So one of the most important things that we figured was that we wanted a virtual team of volunteers. This means that you weren't conscripted into on call just because you happen to be a member of a specific team. You would just nominate yourself as being ready and willing and able to go in Call for a shared environment and take OnCall take pages from any part of the company.
So this was satisfying. It meant that people could select in. It meant that people who didn't like on call wouldn't be didn't have to do it and wouldn't be scared away from working with the teams where some of our biggest problems were and happened to have a high on call burden.
So this seemed to work well around the engineering fluidity and also gave us flexibility and the ability to use it as a growth opportunity for people. We also figured out or wrote standards and processes for bringing in and adopting alarms into this environment.
So we use technology to solve some problems here. We we use a mix of Datadog and CloudWatch to to to manage our paging and alarming stuff. So we we set thresholds in those tools, and they get fed into PagerDuty, and that's what bothers us on our phones.
But they're all kind of configured by hand, and we were kind of missing a lot of process or any process really around setting thresholds and figuring out why things were set, whatever. So we've been doing a good bit of infrastructure of code work at the time and figured that we could build a Terraform pipeline to apply basically peer review and audit logs and history to our alarming setup.
Having all of this configuration in one place, we figured would be a good thing for looking at the overall usage of our alarms and runbooks and stuff like that. We also stated that every single alarm must have a runbook. Sometimes these runbooks could be shared between alarms, generally it wasn't good enough to just have something that would page and we could no longer expect that the person who was on the other end of the pager would know exactly what to do because the volunteer could be somebody of any team
who's never experienced working on that team whatsoever. So we moved the moved alarms basically one by one over to the new virtual team. So we set up a new PagerDuty rotation, put people into the schedule, and progress here was actually a bit slow.
We did get good support from engineering leadership, and teams were kind of given deadlines and goals to get the alarms over. And what worked was we sold the story pretty early on to engineering leadership while we were doing it and so we got unwavering support and so ultimately getting alarms over into the team was made an engineering priority and so this work was real work that we wanted to succeed and we wanted to happen across the entire organization.
So sometimes you need a bit of stick as well as character. One thing we didn't want to do was break the strong ownership and strong linking between teams and their services. So I think successful DevOps teams and successful engineering teams really need to have a strong connection between their services and what runs in production.
So what we figured was our out of hours what we didn't want to fix or threaten was that linking between them. And so by just addressing out of hours on call, which to be honest is when things tend to break less because people aren't in the office shipping code and doing things which tend to break things, we were moving the minority of alarms and the kind of annoying alarms out of the the away from the service teams, but they would still deal with their own alarms during their
own office hours. And, yeah, so because we that's when we shift things the most, that's also when when our traffic ramp up happens to occur because we're a we're a business to business SaaS company basically that largely sells in Europe and the USA.
So our traffic really ramps up starting when Europe starts to wake up and kind of peaks just when when the West Coast are starting to get up. So this all happens nicely during office hours when you're in Dublin and London. So we didn't dilute that ownership thing as much and most alarms still fire during office hours.
Did this like well, we ended up expanding. We were solving just for Dublin based engineers at the time. We only had the Dublin office in Europe and the San Francisco team weren't as big or didn't own as much parts of the product as they do now.
So, yeah, so that's what we implanted. It had a team. Got to make sure that the on call team themselves got the pages outside of errors, but the team themselves got us inside errors. And this made sense from the traffic and ownership perspective.
What we end up implementing, like, you can kinda go this is kind of like an arbitrary decision, but we figured that we're gonna put somebody on on call for an entire week. So you would start on Friday evening, you do the weekends, and then that would your shift would carry on all the way until the next Friday.
There's loads of ways you can kind of chop this up, and we thought about separating out weekends as being distinct from the weeks themselves or maybe having the week so that person has some context going into the weekend. But this made sense for us, so we went back and forth a lot, but ultimately, you just have to decide a single way to do this.
We figured that we wanted about six or seven engineers on the team. So it's a bit of an anti pattern if you're brought in to do one call once every year or six months or something like that. You lose some context, you don't have the understanding of what's been breaking recently or the kind of things that can be a barrier to being effective on on call, so access to different tools or these kind of things can get stale as things change.
So we wanted people who weren't doing on call enough to get burnt out, but that were doing it frequently enough that they wouldn't lose the entire context and it wouldn't be as stressful when they're doing it the next time. So yeah, six or seven engineers, six months on a team.
We wanted to bring people in and rotate them out over time. And our thinking as well here was that the on call engineer just kind of stops the bleeding. They're not experts for all parts of Intercom. They don't know deeply necessarily any individual part, they know enough context to follow the runbooks and have a general understanding of what's going on.
But when it comes to say specialist knowledge of a distributed system is going really crazy or there's something deep product consequences that where you'd clearly need more people involved to do things, we did want to escalate and make sure that the responsibility of the on call engineer is just to do CPR, do the triage, and then get other people involved if necessary.
And so then we have an escalation chain via an engineering leader who takes a lot of stress off our on call engineer. Escalate. They can figure out which teams or which individuals to bring in. They've got product and business context. They can make the call that we don't need to respond to this alarm, that we're comfortable letting this thing be broken until business hours.
So we didn't want our on call engineer having to make difficult business calls, stressful enough being on call anyway. Also, previously, the on call setup was semi best effort, was unpaid in some cases for teams, and so we put in place a simple remuneration scheme where you're paid a fixed sum per week on call.
It's not life changing money, but it's a nice sum that shows that we put a bit of value into this work. The escalation then escalates to the engineering team manager and if needed, we bring in individual engineers. So this is kind of the design of the escalation thing that we figured out could work.
And yeah, you're remunerated whether you're paced or not. One thing is that because we're a virtual team as such, we're not sitting beside each other day to day, so it's hard to tell if somebody is fatigued or and people reacted getting woken up at night and in different ways.
So we made sure that when you were being brought into this team, it was clear that it was your responsibility to kind of make sure you manage yourself, that if you need rest, if you need time off, if you you just dealt badly with a call in the middle of the night and couldn't get back to sleep, you kinda need to let us know so that we can swap people out and do things with it.
So this was a responsibility of the person who's been on call that we kind of we do want them to have a good experience, but we do hold them responsible for minding themselves. So that was the design, but a bunch of unexpected stuff happened.
So the escalation chain I mentioned of going kind of up through the leadership chain, then backing it down into teams actually became pretty uncommon. What we ended up seeing was people would be like, an alarm would go off. People would be around on Slack, typically, say, people working in the SF office, say it was on the evening during the week or whatever, and people would just chip in and help out.
And so if the on call engineer got stuck, there would be enough people around to just kind of help out and just we didn't follow that formal chain and we were able to figure out what to do a lot of the time without going through the official process.
Our teams took very strong ownership of alarms at cause pages, so it's kind of the case that we discovered that it was the kind of case that nobody wants for people to get people outside their team out of bed. Teams tend to tolerate it a bit more inside their own teams, and so we never had to, say, hand back alarms to teams.
That was kind of in the design of when we were taking alarms over. We opened a high priority issue for every single alarm that fired, and teams were, like, almost embarrassed that their stuff ended up causing a page for somebody, like, outside of their own team.
A few things happened that we didn't which were very positive things, which weren't necessarily part of this design. Because we went through the process of importing all of our alarms into Terraform and writing runbooks and basically applying, looking at what a lot of these alarms were, we figured that a lot of the alarms weren't very good, that they might have been put in as point fixes after individual events, but that in a lot of cases, we've got higher level alarms on real customer interactions or closer to the customer that can give a signal whether
something needs to get looked at. So we ended up throwing out I think something in the order of two thirds of our alarms by just looking at them and evaluating them against the the kind of quality metrics that we had produced as part of our documentation.
Retaining people in and out of the team worked, we kind of worried that it was gonna be difficult to recruit people into the team and that we would have to lean on our systems engineers a bit more. But actually a lot of my job organizing this team over the last couple of years has been trying to manage people coming in and making sure we got a flow of people going in and out that works because, yeah, it's like a really popular role for people to to do
in at Intercom. So this this was a nice problem to have where we're vastly more successful and popular than we thought we were going to be going into it. Additionally, the number of out of hours pages dropped significantly. So because of the continuous improvement we're applying, like every single page got issue opened, we had multiple eyes on them and the social stigma around sending alarms to people out of their teams, we saw month on month consistent dropping of alarms.
And we have spikes from now and then. Things change, we're still growing a lot, we're still building a lot of software, and things can just kind of randomly break. So it's not always going to be it's not necessary to the success of the team, that it's always very, very low.
What we want to see is that our process is being consistently applied and that we're doing the right thing when it comes to evaluating our look in alarms and giving the right feedback to teams. So this has been pretty it's been reasonably consistent.
We had three pages last night, which was kind of frustrating. But I was on call the weekends just from Friday to Monday and I had zero pages, which pretty nice. So has been good and successful. It's not critical to the overall project. I'd still do it even if we still had higher alarm counts.
So I mentioned our engineers in San Francisco, and we had designed this purely for our Dublin teams at the time. But as Intercom grew, we started to do more research and development out in San Francisco. Remote work is hard and virtual teams of engineers involving remote engineers is probably even harder, but there were a bunch of enthusiastic engineers who just kind of wanted to help out and would see things break and would help out on individual issues, and then naturally wanted to get involved in the hours of
hours on call. And as we were building the R and D teams in San Francisco, we did want the engineers there as well to have a good experience of not having to do on call and all that. So we ended up having to change around our setup so that we're with twenty four by seven effectively, that SF engineers can or teams can hand over alarms to Strike or to the virtual team, and that means that Dublin based engineers would then be on call inside of office hours before SF stuff, and vice versa.
SF engineers would be on call for Dublin stuff in SF office hours. So it was kind of not by design, but it was great to see, and it ended up being a pretty good way of expanding the team and also building good connections with folks in different offices.
Later on, our London based engineers have been involved. So we don't have the same time zone problems, but again, as we've built out R and D teams in London, it's been pretty natural that they've been getting their alarms over to the setup, and also the engineers there have been getting involved as well.
So we were already well set up to have remote people involved. So this was definitely not hard to do after we had already expanded it to SF. Another thing which was successful has been regular monthly communications to the entire org about how things were going.
We get good feedback from engineering leaders on this, and people can see that the success of the team is going pretty well. Also, we do anniversaries and performance reviews and then we celebrate work that people do in Intercom, and membership of the team kinda comes up a lot in there.
So this has been, I think, a good part of how it's sustainable and is valued by the organization. Additionally, we let we built a global diverse engineer led virtual team. And the diverse thing is interesting. We didn't design this from the start such that, you know, that was a goal that a lot of, say, women engineers would get involved.
But over time, it's just worked out that way that because it's an open and accessible team and people see it as a way to, like, show growth and leadership and ownership of Intercom, It's been a pretty and it's self selecting into it. We've ended up with a diverse team, so it's not just full of cranky old systems engineers like myself.
We also I was able to write a cool job spec for systems engineers, which meant that we I could say that we didn't have to you didn't have to do out of on call to just to do systems engineering intercom, which is pretty satisfying because there's a lot of stigma and it opens up the role to to a more diverse set of candidates of people who don't necessarily want to do Ed of Arrows On Call. So takeaways.
So challenging every part of your On Call setup, I think is doable in every organisation, just looking at how On Call is done, the practices and procedures and policies and all this kind of stuff, even the technology stack. I think it's doable anywhere and you can really improve the quality of your engineers' lives by doing it.
Be radical when it comes to making decisions about this area. Turns out that humans are more important than your computers. That does mean you have to understand your customers and your SLOs, but your on call setup should really optimize for your people rather than your tech stack.
Because your humans are greater than computers. You can fix computing problems with humans and typically not the other way around. I think deleting a lot of your alarms is a good thing. I think in most cases, we've also gone back after having the on call setup and just periodically reviewed what's gone in there, and turns out just with change and with growth, a lot of alarms can be removed even with a good process where we do a peer review into it.
So continuously improving on call buys time for your teams and focus, and don't let on call eat your engineers. Thank you.