Sally will discuss what technical debt looks like, and how easy it is to get into debt in the first place. Then she’ll put a plan into action, including: Facing up to all the debt you are in; Deciding How Much You Can Pay Back Each Month; Getting help when you need it; Being Diligent Moving Forward.
← All talks 
Build Turing Fest 2022
How To Get Out Of (Technical) Debt!
Auto-generated transcript - may contain errors. Tap a timestamp to jump the video.
How many engineers are there in the room? Oh, most of you. Okay. Interesting. I usually am wearing my reading glasses to look at speaker notes and I don't need them today and usually when I wear my reading glasses, I can't see anybody and now I can see you all and it's terrifying.
Anyway, thanks for coming. Technical debt. Engineers cry about it, work on it secretly. Product managers' eyes glaze over when you talk to them about it. CTOs spend sleepless nights worrying about it. Nobody wants to talk about it. It's the elephant in the room.
Nevertheless, we're going to talk about it because the aim of this talk is to convince you that everybody in your business, not just your engineers, should care about technical debt and everybody should care about reducing technical debt and you need to care about reducing it.
And I'm going to talk to you about some sustainable ways that you can reduce it. And this is what we're talk about. We're going to talk about first half, we're going talk about being in debt. And the second half, we're to be a bit more cheerful and we'll talk about being out of debt.
First of all, let me introduce myself. I'm Sally. I'm an engineering manager. I like to swim in stupid places. For the last three, I've been a manager in tech for about eight years and the last three companies I've worked at have been startups.
And each one of those companies when I joined them was less than six years old. And each one of those companies was kind of working in a really, really competitive market to get product market fit, working to kind of beat their competitors, working super fast.
And each one of those companies, even though they were less than six years old, were really, really constrained by the technical debt, massively slowed down by the technical debt. And the last two companies that I've worked in, I've been given the role, it's been mounted on my desk, of trying to kind of figure out how we can kind of manage our technical debt.
And I'm going to talk about some of those kind of stories of how we've managed to do that in the second half of the talk. This is the company I work at, at the moment. We're Accurex, we're a medical tech startup. We work to improve communication between users in the NHS and patients.
Since COVID, we've had a pretty busy time as you can imagine. We've gone from thirty percent adoption in GPs practices in England and Wales to ninety seven percent adoption in that period. And as well as growing massively in scale that way, we have also created the COVID vaccination booking system for GPs. So it's been a pretty fast couple of years for us.
I joined the company in January of this year and literally the first project that landed on my desk was like help us manage our technical debt. So let's go into the first part, what is technical debt? Some examples and how do know when you're in debt.
What is technical debt? Technical debt implied costs of additional rework caused by choosing an easy limited solution now instead of using a better approach that would take longer. And what does that really mean? How many of you have got student debt? Yeah, right.
How many of you when you first decided to go to university or college, said, I know, I don't want to go to college now. I think I'll wait for a few years and save up the forty thousand pounds and then I'll go to university when I've got all the money.
Yeah, nobody, right? Because what you wanted to do is you wanted the thing, your university education now. So you went to university, you took out the debt on the understanding that you'll be paying it back over the next few years. You borrowed from your future self.
Same with tech debts, right? You're building a product now. You want to build it fast because you're working in a competitive industry. You make certain trade offs. You know that you're going to want to ship fast, so you make certain trade offs. You know that you're going to have to pay them back later.
And the paying them back later in the kind of fixing is your technical debt. But of course, with technical debt as opposed to with financial debt, you're going to be kind of creating the debt all the time. Every feature you build, you're going to be accruing technical debts.
And the last time I gave this kind of similar version of this talk, somebody said, Oh great, I work on kind of legacy code base. I but I'm building a kind of greenfield project and so I'm coming to your talk because I want to know how to not incur any technical debt.
Not going to happen. Sorry, if you think that's going to happen in this room, you're absolutely wrong. I'm not going to tell you how not to incur technical debt. I'm going to tell you how to fix it as you go along. Because in fast moving startups, it's going to be worse, right?
In super fast moving startups, The faster you move, more to be more competitive, the more experimentation, the more change, the more unpredictability, the less time you have to think, the more you're going to accrue debt as you go along. And the less time, in fact, you're going to have to pay back your loans over time.
So accruing technical debt is inevitable. It's an inevitable consequence of wanting to deliver fast and beat your competitors. It's a trade off. Does fast mean bad? I'm talking about being fast. What I'm not saying is there's a whole bunch of engineers just sitting there bashing away, creating bad code.
That's not what it's about. Because if you built your software perfectly, you'd go out of business. Okay, I'm going to give you some examples of technical debt now because people talk about technical debt, they think it's just code debt. And there are many different types of technical debt out there, all a result of moving fast.
So let's just have a quick look at some of them. Code or application debt. So code that's overly complex, inconsistent, has duplication, difficult to test, hard to understand, poorly documented and the consequences of this can be development or maintenance is harder and slower, engineers can be demoralized or confused.
Design or architecture debt, not thinking about the architecture of your systems and not adapting to change as your systems change. Consequences of this can be your systems are so complex, can't be adequately managed, you have single points of failure. Testing date, debt. So debt, that is testing your unit tests, your integration tests or your end to end tests are lacking or there's kind of bad balance of those.
And the consequence of bad testing means more bugs, means lack of confidence in your ability to ship, which can mean that you can bundle your kind of features more, which can in turn make more risk and can add to kind of more bugs in the future.
Tooling debt. So this means libraries, frameworks and tools not being kept up to date. And the consequence of this means you can't take advantage of new features in those tools and it could also mean really bad things in terms of your tools and libraries become end of life and not supported, which can have big problems for security.
Reliability and performance debt. This can mean not kind of spending time and effort on improving your reliability and your performance as you go on, which can mean degradation in your performance and bad customer experience. Knowledge debt, so lack of documentation, useful documentation. The consequences of this are you can't onboard new engineers fast enough, junior engineers are less confident in making changes because they don't know your styles and skills there.
So this could be too many two junior engineers in your company or too many new engineers in your company and the consequence of this combined with the knowledge debt means that you kind of can't work at pace and your engineers become more demoralized and your more tech debt is incurred more quickly.
So this is kind of just a kind of snapshot of different types of debt. Don't just think it's code debt. And why are these all important? They're important because they're all a consequence of moving too fast or not kind of taking the time to kind of look at your decisions as you move along.
How do you know when you're in debt? Excuse me. When I said I was going to do this talk, I asked people what they wanted to know and they're like, how do you measure technical debt? So I kind of went onto the internet to do a bit of research and started kind of Googling and I kind of found all about kind of psychomatic complexity and code coverage and the SQUAIL method and I was like, What's the SQUAIL method?
I have no idea. I don't even know if I'm pronouncing it correctly. And I kind of started, I was kind of thinking, Okay, cool, cool. I could do this. I'll make a few slides about this. And I kind of thought, You don't need to measure cyclomatic complexity, right?
You just need to kind of look and see because the manifestations of technical debt are all about you. I'll tell you what I mean. I'll show you what I mean. Let me introduce you to Leon. Leon is a junior engineer. Leon has been asked to kind of work on a small feature, a very small feature, and he's been working on it for around a week.
Every day at stand up he says, Yeah, yeah, yeah, I'm almost done. I'm almost done. At the end of the week he ships this very small feature. His tech lead wants to kind of have a one to one with him to find out why it took him so long, thinking there might be some kind of learning opportunity in this for him.
So he sits down with a tech lead and he says, Oh no, no, the feature only took me a day to write. That was fine. It was the unit test. The unit tests were a nightmare. One unit test took me three days to write.
It actually happened to me earlier on this year. That is tech debt. Let me introduce you to Anna. Anna is a senior engineer. Anna has been working at her company for four years since there was a small startup, very competitive landscape, moving really fast.
Anna was responsible for writing most of the code for the last four years from when she was quite a junior and inexperienced engineer. And now she's paying the consequences of moving fast and making some questionable decisions in coding and design. And all of her team also having to kind of deal with the consequences.
Now Anna, you know, is a kind of great engineer and following the boy scouts code, she wants to tidy up after herself but she hasn't got any time to do so. So every time Anna is estimating a feature, she adds two days onto the estimate and she quietly refactors as she's going along.
That's tech debt right there. Let me introduce you to Christina. Christina is the user help manager. Christina is having a really, really bad day because today she had a whole barrage of frustrated users contact her because a team has shipped a feature which has caused the critical parts of the system to fail with a bug and the users have been complaining.
So Christina has gone on to Slack to try and kind of contact the team that released the feature that had the bug and teams nowhere to be seen. They're in the pub. Christina is wondering why a team would have released the feature and then gone off to the pub.
And after a bit more investigation, Christina finds out that the team that in the pub weren't responsible for the feature, that it was a completely different team, released a completely different feature in another part of the code base, another part of the product that had the consequence of breaking the feature over there.
Neither of those teams realized that they had some code in common and that one would influence the other. That's tech debt right there. Anybody experienced any of these situations before? Yeah, of course you have. I've seen them all in the last year. You don't need to measure cyclomatic complexity.
You don't need to know what the SQUAIL method is. You just need to listen and see and then you'll know that your team is in tech debt. Your engineers know it, your product managers probably know it as well, your CTO knows it and your business leaders know it.
Because tech debt is not a technical problem, it's a business problem. You know that you've got a problem when the speed of your delivery slows down, you can't estimate properly, your deadlines slip, your rivals ship features faster than you do, your reputation is tarnished because you're releasing bugs, you're having outages, your customers become annoyed and miserable and your security is compromised.
You experience attrition. Your engineers don't want to work on your code base. They're demoralized and frustrated. New engineers are difficult to onboard, there are whole areas of the code base that nobody wants to work in and it's difficult to retain talent. You're in debt and you know that you need to take action.
Okay, second half, fifteen minutes left, that's pretty good. Okay, we're in the second half. This should be the happy bit. How to pay back your debts. We're going talk about how to pay back your debt first. First thing you need to do is face up to all the debt you're in.
So when you're in financial debt, what do you do? You write a list of everything that you owe. It's exactly the same with technical debt, advise you. Get everybody in your organisation to input into a document, write everything you think that is wrong with your system, put it all in one big document.
Everybody will have different perspectives to get everybody involved. The second thing that we've done basically is have a bit of spin. Words are important, I think, and if you ask everybody if they want to work on tech debt, everybody in your business, most people will just look a bit miserable.
We rebranded our tech debt project to be tech investment because if I said to you, Do you want this one hundred pounds do you want to spend it on debt or do you want to spend it on investment? I'm sure most people would say investment.
So we rebranded our project as a tech investment project. Okay, managing repayment. So you've got your long list of tech debts, tech investments. You've rebounded your project. Now you need to pay back your debts. How are you going to do it? I'm going have a look at these different methods of like repaying your tech debt.
Never want to stretch a metaphor to breaking point. We're going look at one off repayments, dedicated teams, ad hoc repayments and sustainable ones. Okay, one off repayments. This is kind of the idea of like a debt bash basically. So everybody in the company downs tools, you spend like, you know, a few days or a week or something working on your tech investment.
Pros and cons of this approach. It's good for kind of fostering a bit of kind of team spirit, right? We're all working on the same thing. We're to kind of fix a few issues. It's good to have a sense of togetherness. You get some good cross team collaboration and you can kind of get a fair bit done if you have a kind of debt bash, right?
So everybody's doing no product delivery, you can get a lot done. The cons though are you can only do this sporadically. So even though you might get a lot done, in reality no organisation is going to do this more than once or twice a year, right?
You can spend time with everybody in the organisation just not doing anything else. So really you can get a bit done but it doesn't happen very often. Dedicated teams. So this is where you might have a team in your organisation, you're like, you're not going to do product delivery, you're just going to sit here and do some tech debt for a few months, take them out of the normal flow.
So you might do this with engineers in your organisation or you might just get some contractors in to come and do some tech investment for you. Pros and cons again. Doesn't interrupt the flow of your normal product delivery. That's good. There's no context switching for the engineers involved.
They can just focus on one thing and you can deliver kind of larger pieces of tech investment if you do this. The cons are actually nobody really wants to do this. Engineers go and work in product organizations because they want to work on products.
So you might have somebody who will work on this for kind of three months or something, but most people don't want to be left in the team doing tech investment. Neither your engineers or your contractors in fact. It uses a lot of your senior engineers.
So tech investment and tech debt generally is complex. You're not going to put any junior engineers on a team that do that or not many of them. So it will kind of use a lot of your senior engineers. And then your engineers who are not working on the tech investment don't get exposure to the pain of tech debt and are more likely to do more of it.
And finally, if you have a team of contractors, they don't know your code base, they don't know the problems, it's inherently a complex piece of work, you're going to have somebody else from your organization who's defining the work, who's reviewing the PRs and so you're going to be using people from your organization anyway.
Ad hoc repayments. This is where I was when our company was when I joined Accurex. We basically said to our engineers, here's our long list of tech investment, pick some up when you can, collaborate with colleagues who aren't in your team. We were kind of fairly kind of laissez faire about it.
The good things about this was it was kind of flexible and efficient. So you might have an engineer who's like waiting for PR to be reviewed or a team that's waiting for some discovery to be done. They could kind of pick up some tech investment and just do it in their own time.
It kind of helped a bit with cross team collaboration because people would pair with people from other teams on something that they cared about. But in fact, there were more cons than there were pros in this approach. So we found that, you know, product delivery quite rightly trumps everything and so engineers would always much prefer to deliver a feature than they would do some tech investments.
So we just found that the tech investment was getting pushed and pushed further down the stack. And on the team collaboration, we just found that there was scheduled clashes. So we wanted two engineers to work on X, Y or Z and they could never kind of find kind of diaries that work together so it didn't really happen.
It was a lack of transparency and that was a kind of really big thing for us. The product managers were saying, what are your what are the engineers doing in that time when, you know, in the downtime? They didn't know. And some tech debt was or tech investment was just too big to be worked on in this kind of ad hoc way.
That wasn't working for us and which is where I came in. So we decided we wanted to kind of make sustainable repayments of our tech investment. And what that meant ultimately was we took our tech investment and we decided that every team in our organization was going to work on tech investment alongside our kind of product delivery every cycle and every team would be accountable for achieving their kind of tech investment goals every cycle.
How do we do it? Got buy in from our senior management, embedded in our teams, empowered the teams to deliver and celebrated success. So first of all, we wanted to get buy in. That was really, really difficult actually. It seems like it's kind of quite a straightforward thing, but it wasn't.
We needed to convince our senior leadership to let us do this. Our VP of Engineering was on board with doing it, but certainly our CTO and our Chief Product Officer and our CEO needed convincing. So we wanted to describe the problem in business language and we did what we talked about, I talked about earlier on.
We talked about the kind of key business problems, which were kind of slowness of delivery, bugs and outages, and attrition and kind of lack of motivation of engineers, and the problems that they would cause. And then we showed them the scale of the problem.
So we had that big long list of tech investment and we'd also we had estimated how long it would take for every piece of tech investment to be done. And we also estimated what the cost to the business would be if we didn't do that tech investment.
We took those kind of two problems to our senior leadership and we managed to convince them that it would be a good thing to embed it in our teams. Then we had to agree how much time that our engineers would be allowed to spend every cycle doing tech investment.
We started with a bit of a softly, softly approach because we didn't want them to say no outright. So we asked them if we could spend ten percent of engineering time per cycle on tech investment and ninety percent on product delivery and they said yes.
In reality, we wanted it to be twentyeighty, but were a bit of cowards to be honest. We wanted to bring our product managers on the journey with us. That was really, really difficult because even though we managed to kind of tell our convince our senior leadership, product managers are the ones who are on the hook for delivering value to our customers and they were really, really concerned and reticent.
We did two things to try and convince them. The first was we reminded them or we told them that they would be allowed to deliver ninety percent of their kind of product work. So we were taking ten percent of engineering time away. We didn't still expect them to deliver one hundred percent of product features.
So we kind of descoped the product delivery to ninety percent. And we said to them that we wouldn't give tech investment to teams that were kind of on the critical path for delivering something that cycle. Next thing we want to do is embed it in the teams.
That meant we wanted to go into a kind of cycle of planning. Just as so for us, the cycle is two months. So just as we have kind of product planning every two months, we wanted to go in and do tech investment planning every two months as well.
How do we do that? First thing we did was get the right people in the room. So our VP of engineering, our technical directors and any kind of other subject matter experts that would have things that are in the list of tech investment.
We've got them all in the room in the planning. We prioritized the work and that was kind of, we used kind of various kind of rules. So some things that were obviously need to be prioritized above others, tools that were coming to end of life, areas of the code base that were kind of worked on much more frequently than others that kind of needed refactoring and areas of the kind of code base that kind of had consequence of which were kind of user problems.
So we had some areas that were kind of more buggy than others that were causing problems for our end users, areas that we couldn't monitor well. They need refactoring so we kind of ordered our lists. Then we sized the cycle worth of work so we kind of estimated every item on that list of priorities.
We knew how much engineering time we had, so we had like, you know, ten percent of engineers times, times X number of engineers, times a cycle And we worked out how many of those tech investments we could do. We drew a line underneath them.
And then we wanted to match our tech investment to our team. So usually a product team will come up with what product they want to work on. We had this kind of big bucket of tech investment we wanted allocated to teams. We had our list of tech investment, we had our list of teams, we had a list of the people in the teams.
First thing we did was like we looked at urgent delivery deadlines of any of the teams. If any teams had delivery deadlines, we didn't give them tech investment. We looked at the skills in the team, so whether there were a lot of senior engineers or there were no front end engineers, and we allocated according to that.
We aligned some tech investment with teams, so some refactoring would go to a team that was working on that part of the code base. And then we looked at the development goals of the individuals in the teams and we tried to align some tech investment with engineers' interests and also their development goals.
So if we had somebody who wanted to be a senior engineer, we'd give them a kind of piece of complex tech investments so they can improve their seniority. Then once we kind of allocated all of our tech investment to our teams, we kind of sense checked it with the product managers and the tech leads of each team, which got us to kind of talk again about why we were doing it.
And then we empowered them to deliver it. And we trusted them to do it. So I'm just going through this fast. They said to us, how do we do? When do we manage it? We were like, it's your tech investment. You can kind of figure out when to do it, whether it's the start of your cycle, the end of your cycle, we don't mind as long as you deliver it.
And we wanted to hold our teams to account and what that meant for us was within a cycle in Accurex, we have three check ins with senior leadership. So the teams report back on their product goals. We made sure that in those three check ins in the cycle we also reported back on our tech investment goals to our senior leadership.
Reinforced the fact that the senior leadership cared about a tech investment, which then in turn gave the teams permission to work on the tech investment, was a really important part of the process. And then we celebrated success. Just as you would do with a product launch, we would make sure that in Slack and in our company All Hands, we talked about when we shipped a very good piece of tech investment, which helped us bring investment out of the shadows by celebrating it visibly.
Has it worked? We've managed to get through a fair bit of tech investment. Our teams have embraced it and so you get a product manager saying, Why aren't you doing that refactoring? Rather than Why are you doing that refactoring? Which is music to my ears.
Engineers don't feel guilty that they're working on tech investment, which is great. There's transparency, so product managers will know what engineers are working on because that key thing I didn't mention was that we put all of the tech investment tickets into our product backlog.
So they're all very visible. Everything that people are working on is very visible. There's a lot of transparency and it helps with the career progression of our engineers. What hasn't worked? It feels a bit top down still. It's still a kind of group of people who are allocating the tech investment.
We could do better with that. And there are kind of a bit of a weird thing with kind of mid cycle moves. If somebody is doing some tech investment moves to another team, kind of the tech investment gets orphaned slightly, but those are kind of minor problems.
How to avoid accumulating debt. One minute sixteen to go. Have some good habits and these can be, be aware of the compromises you make. So if you are kind of choosing fast over perfect, sometimes don't always kind of go that way and like try and kind of redress it slightly.
Don't always make the kind of the fastest choice you can. Document your trade offs. So if you are working super fast and you know that you're making a bunch of trade offs, write them down either in a document or in the code so that you can come back and revisit them because you'll forget about them.
Use Lean UX methods. So if you, you don't always have to build something. If you're doing a lot of experimentation with your customers. You can use kind of lean UX methods to not build things before experimenting. Invest in knowledge, write things down. Don't always just think that you're going to do it later.
Upgrade your technologies, have a spreadsheet of all of the versions that you're on when things are becoming end of life, set alarms, somebody's nodding. Keep your tech investment list up to date so you always know and go through this process every cycle. And reward continuous improvement.
We've put tech investment improvements into our career progression cycle framework so that engineers know that we value it. Conclusion. Phew. Minus seven seconds. Oh my God. What do we talk about? What technical debt is? Some examples of technical debt: visible manifestations of technical debt, the business impact of debt and sustainable ways of reducing technical debt.
But if you remember one thing from this talk, I'd love you to remember that to treat technical debt as a business problem, not a technical problem. Leon will thank you, Anna will thank you, Christina will thank you, Your business leaders will thank you.
Your customers will thank you and your investors will thank you too. Thank you.