In this talk, Mike McQuaid walks you through some heuristics to build better software, quicker by learning from the past, the present and the future in your software projects.
← All talks 
Build Engineering Turing Fest 2022
Ignorance, Incompetence and Insignificance: The Ingredients To Build Great Software
Auto-generated transcript - may contain errors. Tap a timestamp to jump the video.
I'm going to talk today about what I think are three important ingredients in software, which are ignorance, incompetence and insignificance, and how I think they help you make great software. So we've had a wee intro for me already so I won't dwell on that too much.
I'm Mike, I'm an engineer based in Edinburgh, I'm from Edinburgh originally, I've kind of bounced around and been away and come back and stuff like that. And I'd like to get some almost like intros from you as well. Don't worry. I'm not gonna actually get too many people to speak or anything.
But like, I just want a wee show of hands of the folks in the room. If you identify as an engineer, can you stick your hand up in the air? Sweet. How about any designers in the house? Yeah, a few folks. Product managers?
Marketers? Any other sort of manager? And anything I've not said? Person who said probably, what is your title or role or what you identify as? Probably other. Oh, there's probably other people. How disappointing. I might my attempt to pick on someone has dismally failed.
Oh, wow. Well, yeah. So we've got a good mix of people here, and that's one of things I like about Turing is that it's a conference where we can kind of come together, kind of cross functionally, and talk about what makes things work together.
Because obviously, if we're building software, we're not doing it by ourselves as much as we may attempt to do so, but it's something which takes all those roles I mentioned and more. So my role at GitHub right now, I'm a principal engineer in our communities department.
We're focused on kind of the open source communities that live on GitHub, and we basically just try and build things like discussions, GitHub sponsors that sort of interface with those communities. My personal role involves like jumping around a bit and sort of putting out fires and being asked this project over here is doing this, what do you think about this and whatever.
I'm kind of doing a little bit more work on a single project right now, but generally it's a lot of like having my finger in lots of different pies. And then speaking of that, like my sort of outside of role work main hat I wear is I'm the project leader of Homebrew, which is a package manager for Max and somewhat bizarrely, some of you may think for Linux nowadays as well.
And I've worked on that for over thirteen years. And I sort of wear the hat of kind of sort of the like technical lead and the person who's been around for the longest, guess, and has the most context with Umbru. But also, like, I sort of acted a bit like a product manager there in terms of trying to decide what features we build or don't build or whatever.
Like, I guess, an interaction I had this week with someone asking, you know, if I'm open a PR for this, will you accept it? And it's like, well, it depends. If the PR looks like five lines, then maybe. If it's five thousand lines, then probably not.
And it's kinda hard getting those types of balances. But anyway, so through all this stuff, I feel like I've sort of learned and had a fair sense over the years of, like, what projects go well and what projects don't go well. So I'd like to start by telling you about the best project I've ever worked on.
So hopefully, you can kind of learn, bring some of this knowledge back to your own project. So we were gonna avoid the mistakes on this project like the other projects I'd worked on and the other projects that the company had done like this before.
So these previous projects were basically just built by a bunch of idiots. They made really stupid decisions, and basically, they just seemed like, you know, rubbish engineers. So we knew what to do because we, unlike them, were smart and clever, and we're gonna do everything right.
So we were gonna do everything right the first time. We'd seen them kind of, you know, oh, well, we're gonna try and do it this way. Oh, that way didn't work. Oh, we'll try to do this way. Oh, that way didn't work. No. No. No. No. None of that.
Like, we could plan in advance. We knew already from their mistakes and failures what the right things were that we needed to do. So we're gonna do those first time. Because we were such an exemplary bunch of individuals, we managed to convince the leadership team, like, look, this project is gonna be amazing.
Everyone's gonna love it. We've spoken to customers. This is exactly what they want. We need lots and lots of head count, and we're going to plan everything, have a nice, well planned backlog where everyone can go and know what they need to work on.
And we're going build something and deliver it to customers, and they're going to love it. And customers unfortunately hated it. What the hell went wrong? So let's start with the first I, ignorance. Let's learn from the past. So on this project, we knew everything that we needed to do, and we did it.
Unlike those idiots who didn't know anything before. But it turns out, when you think you know everything, you actually know nothing. This is where the troll in the crowd shouts out Jon Snow or something along those lines. So first thing, a slight minor tangent.
A classic thing you see with people who think they know everything is often people as well are like, I have a great memory, which is if someone ever says that to you at work, I would recommend you start worrying. Because generally, people who say they have a great memory actually have a terrible memory, and they rely on their memory to remember what they have to do.
And therefore, they forget almost everything they have to do. And if they say they will do something for you, they will never actually do it for So I have, like, my own little system. You can find it at this, like, short URL for, like, how I get things done.
And step one of getting things done is when you say you're going to do something for someone. Write it down immediately, anywhere. Just write it down, please. Write everything down, everywhere, always. The next thing that I like to think about with ignorance is, has anyone heard of the concept of Chesterton's fence before?
A few people. It's one of my favorite things. I say it all the time at work, and people are gonna get fed up with me saying this so often. It provides a really interesting way, I think, of thinking about ignorance. So I'm gonna read out a little quote from GK Chesterton.
This was from, I can't remember, some early period in the twentieth century, so excuse the slightly antiquated language. There exists in such a case a certain institution or law, let us say for the sake of simplicity, a fence or a gate erected across a road.
The more modern type of reformer goes gaily up to it and says, I don't see the use of this. Let us clear it away. To which the more intelligent type of reformer will do well to answer, if you don't see the use of it, I certainly won't let you clear it away. Go away and think.
Then when you've come back and tell me that you do see the use of it, I may then allow you to destroy it. Now this seems slightly contradictory, but it's something I run into all the time at work, where you get a team or a person how many times have you seen this?
This code's useless. Let's delete it. Let's just destroy it. Or this system that we built is stupid or pointless or all the design constraints that are there because the previous team were idiots. Well, until you can explain to me why they weren't idiots and what they were thinking of when they made those decisions, you probably should not be going and refactoring that code.
You should probably not be re architecting that system, and you shouldn't be deleting everything. Okay. You figured out maybe a reason why this existed, and you can explain that. Okay, I might now allow you to go and change it. But again, make sure you have a decent understanding and make sure that that understanding does not rely on things like because they were wrong or stupid or whatever.
Because design constraints change over time, may well be they were optimizing for a case that you are not optimizing for anymore, it may well be customers behaved in a certain way before, and they didn't behave in this way anymore. And that's cool, but like you want to be able to explain why that is.
But that's not always practical. Like in companies like GitHub, I'm sure those of you who have been at a company for a little while have seen the same thing where you have people in companies who say, well, we need to find the person who wrote this code and ask them why they did that.
They left five years ago and so did everyone on that team who worked on that project. And turns out they didn't write any documentation. Turns out they had all their conversations and DMs on Slack, so we will never know, this loss to the ethos of history, what actually happened here.
So, okay, in that case, you might not be able to explain why, but then at least you can try and be a little bit more careful then. You okay. You might wanna write some work, like, tests to make sure you're confident when you're refactoring this code.
Make small changes, little, little small changes, and be prepared to iterate and learn while you do these things. And most importantly, remember your ignorance. Remember that you don't understand this system and you're changing it despite your lack of understanding, but be careful. So the next I I'd like to talk about is incompetence, which I, in this example, talk about learning from the present.
So again, on that project we worked on before that was amazing, where we did everything right first time without making like those mistakes that those stupid incompetent idiots that I work with are always making because they're not as clever as me. Unfortunately, every time you try and do it perfectly the first time, probably actually going to do it wrong the first time.
And if you're not willing to make mistakes along the way, and even worse, when you're making mistakes and you're aware that you're making mistakes, but your own pride won't let you own up to them or admit them to other people and talk about them openly and have a culture where you can't talk about making mistakes.
You're gonna make big mistakes, and they're gonna be more severe than they had to be, and they're probably gonna be more public than they had to be as well. So a podcast I really love. Anyone listen to this? Cautionary Tales with Tim Hartford.
He's a chap who writes for the Financial Times amongst a bunch of other places, and he basically talks about a bunch of effectively errors in ways of human thinking and talks about kind of some contemporary examples, some historical examples of like patterns of behavior people have fallen into and stories about things that have been done in a certain way.
And most of his stories either look like things are going well and they actually go badly or they look like they're going badly and they actually go well. So this episode is a particular favorite. If you just listen to one, I would check it out called Bless the Cold Black Hearts of the Broadway Critics.
And it used two examples that I like to think about when thinking about failure. So the first is this musical, Movin' Out, the Billy Joel musical. Apologies if I, say anything wrong or, like, slightly confuse things here because I'm neither someone who has much knowledge on musicals nor am I a big, like, person who knows about this musical specifically.
But from my understanding from the podcast and reading, it was basically gonna be a big deal. So the the creator of Twyla Tharp, like, was gonna do this, like, I think it was choreographed very differently to normal, and a bunch of people were really excited.
They were excited about the musical side. They were excited about the kind of the dance side and the choreography side and stuff like that. And as a result, I think fairly unusually, it got reviewed a lot more heavily than it would do in its previews in Chicago.
So my limit on the semi musical world is you might do previews while you're sort of building your show and trying it out, and then you then go to Broadway and do almost like your main production whenever things are ready to go. And you might have a few reviews of the previews, but the national press would generally try and kind of almost like cut you a bit of slack and save the reviews for later.
But in this case, the national press did go and reviewed some of these kind of early previews in Chicago and generally seem to agree that it wasn't very good. And unfortunately, it had to have a huge amount of changes that were made to it.
But the creator could have gone and said, okay, like, this is really embarrassing, but we've got a vision or we're gonna go through or they're wrong or we'll prove them right in the end, but they didn't do that. Instead, she went and looked, got a note of basically everything that had been made, and criticized across these reviews anywhere where two people had come across the same problem and agreed on what it was and basically tore the musical to bits, rebuilt it, like we're continuing to perform the sort of old version of the musical,
the almost like v one at the same time as kind of building and planning and rehearsing for the v two, which on paper sounds terrible. But in reality, by the time they got to Broadway, this musical was a sensation and it won a bunch of Tony Awards.
So to me, I remember reading some of the stuff that she'd been saying, and she very much talked about this concept of failing privately and failing publicly and about how for her, like, there maybe not been enough private failure. There maybe not been enough candid conversations with people before because everyone was so caught up in the idea that this was gonna be a success.
The next slightly different example is this strange looking thing, which is called the Gossamer Condor, which was the first human powered aircraft that won the Kramer Prize, I believe. So human powered aircraft obviously have the problem that they can't rely on fuel or engines but have to rely on the output of a human, I believe, generally always like some sort of pro cyclist person who can output a bunch of energy and doesn't weigh too much.
They're made from very very light materials, they have as little on them as possible. But as a problem that comes from doing this is they are not terribly resilient. This thing crashed a lot. Every time it crashed, it needed to be rebuilt, sometimes rebuilt almost entirely from scratch because, as I said, not very resilient has to be light.
But it didn't win because they built the right thing first time. It didn't win because they sat in a room and planned what the perfect outcome was going be. It won because they built it, it failed, it crashed. It had to be rebuilt.
They built it, it failed, it crashed. It had to be rebuilt. They did that again and again and again, I don't know how many times. But they also managed to, while doing that, maintain motivation and maintain a sense of like, we can do this.
This is not a failure. We are iterating towards a good solution here. And they did. Kept failing and breaking and failing and breaking and failing and breaking until one time it didn't fail and break, one time it succeeded and won this award that no one else could win.
You will fail. Software is the same. Your software will crash. Your software will have bugs. You will probably write those bugs, and you will probably find them later and say, which idiot wrote this? And then realize that in fact that idiot was you.
Users will probably hate the first version of your software. Users may well hate the tenth version of your software. But if you try and avoid failing at all costs and if you're so desperately afraid that if you fail, then it will something terrible will happen to you, then probably what will happen is you're just gonna fail later and you're gonna fail bigger.
Instead, fail faster, iterate faster, and improve faster. If you fail privately, if you have a small group of people in your company who see maybe a first version you built and take you aside with love and care and delicately tell you that this is a terrible idea or a terrible execution or this isn't gonna work, great.
If only four people see something that you've built and then they tear it to shreds, sweet. That was only four people. Think about what would happen if everyone told you that this was a great idea, and then it got released to a million people.
And then instead, a million people tell you this is a terrible idea and that this is useless and no one wants it. Private failure is a really good thing. Private failure is okay. It's not gonna be without its share of embarrassment and maybe a little bit of damage to your pride and stuff like that.
But it's gonna be way nicer than it is everyone tells everyone each other all the time that everything's great, and then you have a big failure on a big public stage. So a way to get around this, I guess I mentioned almost like getting feedback from people internally, but this depends on what type of company you're working at, what type of stuff you're building, whether this can work for you or not.
But at GitHub, something we found really helpful being a developer tools company, which is used in building software, is we use GitHub really heavily in building GitHub. So we do what we call dogfooding a lot, which comes from the expression like eating your own dog food, doesn't sound terribly pleasant, but it's a good idea, honest.
And we do what we call staff shipping features before they get released. So we have a way with our kind of feature flagging system to say if someone is an employee of github dot com, then we show them a different version of, you know, say a pull request page or whatever when they're using it to what everyone else might see.
And this enables us to see and iterate and get feedback from people inside the company who are using this stuff really, really regularly and really, really heavily to find out, like, what do these people think, and is this working the way we want it to?
We can also gather metrics on, like, how that changes the behavior of the people inside the company. And then sometimes, we do stuff like this, and we build some feature, and everyone hates it. And turns out, we had a really good idea, and our customers wanted it, but, like, the actual execution of that idea was awful and worse than nothing at all.
So we don't ship that at all. And there's been various things, like, left on the sides of GitHub over the years where I mean, the fun one was about maybe a year and a half, two years before Slack took off. We had effectively built a Slack competitor inside of GitHub that we were going to publicly release that was like, you know, attached to GitHub repositories or whatever.
But in the course of rolling that out, we realized that it actually wasn't very good compared to the other offerings in the market, so we dropped it. And that project never got seen to the light of day. But a bunch of people tested it and a bunch of people gave some input and that still informed and helped the company learn how we can do things better in future.
As I said before, remember that failing publicly is ideally what you want to avoid because if you can fail privately instead, if you can have a smaller group of people inside your company see things, then that's not going be as embarrassing for you, and it's not going be embarrassing for your company, and your competitors will not see that you've maybe made a misstep in that direction.
It's low cost to fail privately. It costs you nothing except maybe a little bit of pride, maybe a bit of opportunity cost thinking about what you could have built instead. But generally, those failures and the embarrassment is gonna be proportional to the audience size.
I think to remember with software though is that some failures will only happen when the load is at a certain level. So if you have something which will live or die based on the performance characteristics of it, make sure you are aware that like when you release this to users, it may well perform in a very very different way.
So ideally try and get a high system load and try and replicate user behavior before users are actually using it. If you can simulate that, then great. Again, other ways that we found that GitHub to kind of play around with this stuff is that we have the concept with like feature flagging that we can do a dark ship, which is what we call it, where we might have, when you're on a pull request page, there's some, like, hidden part of the page that is, you know,
being rendered or hidden or is pushing some data through the back end or whatever, but you're not actually seeing that yet. So we can still evaluate a certain amount of performance characteristics of the system without changing the behavior of that code yet and without making users have a different experience.
So remember that you are incompetent, but it's okay because you can plan accordingly. And the final I I'd like to talk about is insignificance or learning from the future. So that project I talked about before was amazing because we were gonna solve everything.
We were solving most problems the company had in one big product offering. It was gonna be absolutely wonderful. And because we convinced everyone in leadership that this was a great idea, we had all the needed resources we needed to ship this project and more.
Like, we might be able to even extend the scope as time went on. We've got a huge team. We got everyone we needed to do this. We had loads of engineers, loads of product managers, loads of designers, everyone. And leadership just said, you know, if you need more people, just let us know.
We can pull people off these other projects because your project is so important. And as a result of that, the scope of this project was super big as well. Unfortunately, though, this is when things started to go a little bit wrong because having this huge number of engineers and this huge number of things that we were building at once introduced a bunch of coordination costs.
The team was really unwieldy. It was hard to kind of change direction or move in the same direction even at the same time. Silos started forming in between the teams, Team A didn't like Team B because they were doing the things slightly differently.
And because we had this big scope and this massive convincing leadership that this was the most important thing ever, we had, like, even more scope came and even more headcount came, and it just started to fall downhill like a boulder. And because we had all this, we also have a huge risk at this point.
We've dedicated an enormous amount of time and energy. A bunch of people have really sold the entire company that this is the thing we're gonna do. We've started to talk to customers and say, this is this big thing that's coming down the pipeline.
You should all be very excited. We shouldn't have done that. We should have kept it smaller. The nice thing about tiny ships is they can get to production really fast. If you have a little thing which maybe changes a tiny part of a tiny page but makes some small improvement to users, you know, an engineer or two might be able to build that in a week and get into production.
And then actual customers can test the actual things you're building and provide actual feedback both in the terms of, like, metrics on how they're using it, but also in terms of telling you what they think about it as well. Tiny code changes for these types of tiny ships are also ways to review.
And when you have tiny reviews, generally, those are gonna be better reviews. When people are reviewing ten lines of code, they can pay a lot more attention than they can when they're reviewing ten thousand lines of code. Reviews that are smaller get done quicker and better, which help the team and the project to have a faster velocity, and things that are better reviewed tend to have fewer bugs.
Having a tiny team working a project means that team is probably going to have closer bonds with each other. They're going to have more shared context. They may well have a team where, if you're lucky, everyone on that team knows what everyone else on that team is working on at all times.
They can't really silo as a result. Tiny teams share ownership with each other. They share work, they might share tickets that they're working on because hey, lucky this isn't this big thing that we have to carve up in a hundred different ways. They don't have an in group and they don't have an out group.
They don't have factions and they don't have opposing goals. Tiny scope is nice as well. It forces you to think about what is actually essential. Antoine de Saint Zoupari said, a designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.
You can probably still take more away of the thing you're working on at the moment. You can probably still make this smaller before you ship it to production. Please do that. Keeping this all tiny means less risk as well. If you have a four month project built by two engineers and it doesn't really, like, validate what you went into it doing, if customers don't love it, if you end up unshipping or whatever it may be, it's not that big a deal.
If you have a project with four hundred people that you spent three years working on before anything gets to customers, that's not good. It's a big risk. You get a lot of talk in startup spaces and software spaces or whatever about MVPs. Almost no MVPs I see nowadays are actually MVPs.
Like, your minimal viable product is not very minimal at all. Reid Hoffman, co founder of LinkedIn said, if you're not embarrassed by the first version of your product, you've launched too late. Think about that. Your MVP should be embarrassing. It should be very, very small. Make it smaller.
But also, when you don't make it m enough, then it might not get seen or used by anyone ever. Again, this is something I've seen where the scope in a project grows and the project chugs along for a year or whatever, and then we decide to kill the entire project.
And because of the way it's been built, because it's not been built in a sort of minimal fashion, no one gets to use any part of that project even when there were parts that were maybe useful to people. Whereas if that had been four three month projects, then it might have been that customers could have had three out of the four parts rather than none of them.
Also, if you're in a small start up, it may well be that if your MVP is too big, oh, turns out you run out of money and your VCs don't wanna give you anymore, and now your company doesn't exist anymore. Sucks to be you.
But as much as it sucks to be you, it sucks even more to be me, because I had to work on the worst project, which is a terrible example that you will learn nothing from. On this project, from the outset, all the timelines were super aggressive, and we knew we were gonna have to compromise absolutely everywhere.
We had all these legacy systems we had to build on top of because we didn't have time to do it properly and build them ourselves the way we wanted to. We had to use an existing system that was totally unstaffed and kinda running in maintenance mode.
We didn't know how the system worked. In fact, no one really knew how the system worked. So we had to rely on trial and error to kinda figure out if what things were working the way we wanted them to, and we had to fix things in the existing system just so we could use them in the new one.
Stuff kept breaking. Failures kept happening. Hacks kept getting shipped because that was the only way we had to move forward. We didn't really understand even why things were breaking sometimes, but we didn't have any choice because we had this deadline that we had to hit, and that was that.
We didn't have as big a team as we needed. We were constantly understaffed in engineering, in design, in product work. Like almost everything that we had to do, there wasn't enough people. So what are we gonna do? Well, we just tried to do the best we could.
The team was put together, like thrown together from random people around the company. We didn't know each other. We hadn't really worked together before. We hadn't built that trust before the project started. We were also all new to the problem space. We hadn't worked on this either type of problem or this particular problem before.
But we did manage to get something out there in the end, barely, and everyone loved it. And because I feel like it's not being as mean to my coworkers to kind of point out the projects that actually went well, like this project was called GitHub Sponsors.
Our beta, which we had basically we built in about six months, allowed users to sponsor each other on GitHub. And we had a bunch of organizational impediments to getting there. We had a bunch of teams and departments of people in the company who said, no.
You can't do this. No. This isn't gonna be possible. No. Whatever it may be. But nowadays, people literally live off the money that they get through GitHub Sponsors. A project created in six months by four engineers, an engineering manager, a product manager, and a designer, and a few other people scattered around the company.
And it's the proudest I've been of any project I've worked on personally, it's probably the best team I've worked on before or since because we operated under tight constraints. But because of those constraints and because our scope wasn't allowed to balloon, we had a single shared North Star vision of what the team was gonna build.
And we had a small team of people, good people who all of whom I still consider good friends to this day, who were able to come together and remember that, you know, we don't know what we're doing. We're not better than everyone else.
We can figure some stuff out even when we don't know today, and we got a pretty nice product out of it at the end and some people who are able to use this to literally make a living on. So in summary, assume you're ignorant even when you feel that you're not, and even when you may feel that you're the only one who's not ignorant around.
Expect that you're gonna be incompetent, and plan that you're gonna have to iterate on this, and that failures are gonna be inevitable along the way. But hopefully, failures can be made a little bit more privately with amongst a smaller group of people. And start and aim to be initially insignificant and grow slowly and steadily, shipping early and often rather than starting big, shipping late or never and failing big.
And those previous emojis feel a little bit harsh because in reality, you can think of yourself, instead of being ignorant, you're a student. Instead of being incompetent, be a zen. And instead of being insignificant, just be a little bit smaller. And I hope that you will be well.
Thank you for coming to my talk today. If you're interested, I've written it up in a sort of blog posty type form that you can get at n m q dot lull slash best, and I'm doing like a roundtable later where I think there'll be some q and a stuff.
But you're not able to make it or you wanna ask a private question or whatever, you can ask me either on Twitter or just send me an email. Thank you.