The definition of 'done' for software has been focused on whether or not the software works as it was designed. This was relevant in a world of static software. Today, software is continuous. In this reality what really starts to matter is what our users are doing with that software. The same software can be used for sharing baby pictures and instigating a mob of Twitter trolls. The systems we create generate outcomes - changes in user behavior. The modern, continuous nature of software allows us to build learning loops and determine if what we’re doing is generating the outcomes we expect.
In this talk, Jeff will cover how technology can enable tremendous gains but, when coupled with the uncertainty of human behavior, can often go awry. How can we ensure we’re working on products that actually make our users more successful? And how do we inspire a new generation of designers and developers to consider a new definition of “done” - one focused on positively impacting customer behaviors?
So we know WHY but what about the HOW? In this talk, we’ll explore a framework for how to think about diversity and inclusion, and practical things you can go and do to make your organisation and culture genuinely inclusive. Because that’s the only way to attract, recruit, promote and most importantly retain a wide range of people. And THAT is the only way to have the highest-performing teams possible.
Auto-generated transcript - may contain errors.
Tap a timestamp to jump the video.
Alright. Good afternoon. Come on. Hey. Listen, I'm super thrilled to be back here. This is my second time at this at this conference. It just keeps getting bigger and better. And I'm really excited to have this particular conversation with you today because it's a bit of an if you've read any of the stuff that I've written over the last few years or if you've seen me talk here and there, you know that we talk a lot about outcomes over outputs.
And we're going to talk about that a little bit today. But this conversation that I want to have with you today is taking that a little step further, because while customer centricity and evidence based decision making should be the focus of the way that we develop, especially digital products and services for the people that we serve.
If we do that blindly, we run the risk of taking it too far and potentially causing some negative real world consequences. And that's the conversation that I'm going to of head there walk you through today. So let's get through it because I don't have a ton of time.
Let's start with this. Once upon a time, this is where I worked. I worked if you were lucky enough to get one of these CDs, some of my work went on to these CDs. And this was about twenty years ago at America Online, and we would send these CDs out.
You know, once every six months, we print fifteen million of them. And and this was the way that software was delivered twenty years ago. It was static. It came in a box, if you recall, you went to the store and you bought a box of software.
And at that time, when we thought about technology and how fast we could learn whether the things that we were building actually delivered value to the customers that we were serving, our learning loops were long. We would ship every six months. We'd wait for six months' worth of usage and then ship again.
So we're looking at twelve month feedback cycles. Software adoption, generally speaking, was low. People had to go and buy expensive computers. They had to buy expensive software. They had to load that software onto their computers, and so the impact of the work that we did, even twenty years ago, fifteen years ago even, was limited to the tech savvy and the people who could afford technology.
Back then, the goal for the teams that I worked on was works as designed. Right? We designed it to work a certain way. The engineers built it that way, and as long as it did what we thought it what we what we told them to do, that's how it was supposed to work.
And the measure of success for us was getting the software out the door. That was it. That was the measure. We got it out the door. We had a party. You got a t shirt with the name of the project on the t shirt, and then we moved on to the next initiative.
Now things have changed dramatically in the last really ten years, if you think about it. Software has eaten the world. It's consumed the world. Everything is now powered by technology. Software, certainly our phones are powered by that. My coffee machine has software in it these days.
And if you're lucky enough, maybe even your salt shaker is connected in some way or another. Right? Every single thing has technology in it. And what has enabled this and what has fundamentally changed is how we make the software today. It's a fundamentally different way of thinking about software development.
Whereas twenty years ago at AOL, it was static. We made the software, we printed it on physical media, and then we shipped it out to you. Today, we make it continuously. Right? Software is continuous. There is no end. It's not static. It is dynamic, in fact, and it's these systems that we continue to optimize.
And it's not just our phones, and it's not just the websites that we use. It's the businesses that we run, and we're seeing that be understood more and more by all the companies in the world. This is the executive group chairman for BDVA, which is a big Spanish bank doing a lot of interesting things with technology, and the part up here in orange is the key part.
Right? At the highest levels of this particular financial services organizations, they're realizing that there's no finish line. Right? The businesses that we're building now that are powered by this continuous software are these systems that we continuously optimize. We continue to make them better, that we continue to improve them, we continue to learn how to do that.
And that is a fundamentally different way of thinking than most companies work today. Most companies think like this. They think like traditional car manufacturers, where we're going to put out the next version of the product or the service, which will be the only way for you to get the new features or the new ways of working that our service provides.
Right? And we took this particular model. By the way, this is the greatest marketing gimmick of the last hundred years. There's no reason for the model year. Right? It was invented by Alfred Sloan at General Motors to get you to buy a new car every time.
And we took this model, we applied it to software. Right? Kind of back in the day, We put model years on things, Windows ninety five, Windows ninety eight, Windows two thousand, and then all of a sudden, right, the model changes. We start to apply technology to things that in the past were static, and it starts to change how those things evolve.
It starts to change how we deliver value to those things. All of a sudden, you wake up, and your car can do new things overnight. Right? Things it couldn't do the night before. And it's not just the the cars themselves that can do new things.
It's the systems that support the cars. This article here in the middle tells a story about how Tesla went from a tweet to a shipped feature in six days. So this guy Lloyd tweets at Elon Musk, and he says, listen. Every time I go charge my Tesla in Northern California, some people I think he uses a different word in the tweet.
Some people are are have left their Teslas there connected, fully charged, and they've walked away, and I can't charge my car. Elon Musk, the CEO of the company, can sense this customer feedback. Right? He says, you know what? You're right. It's an issue.
I'm on it. He tweets back. Six days later, Tesla changes how the superchargers work. Every minute that your car spends connected to the charger after it is full, you get charged money for. Right? The way that they're able to do that is because they think about themselves as a software company.
They think about continuous software, continuous improvement, and continuous development. And with this amazing capability, this ability to sense what the market wants and to respond to that as fast as we want, really, as fast as an organization that we want to do that, we have tremendous opportunity.
We've got this capability to see how well our ideas meet the needs of our customers. We can put things into customers' hands very, very quickly, usually the smallest things that we can because these cycles are fast. We can sense how that impacts the behavior of our customers.
Are they more successful? Do they buy more stuff? Do they spend more time on-site? Do they tell their friends? Or do they do the opposite of all of those things? And then we have the opportunity and, frankly, the responsibility to answer the question, what are we going to do about it?
Right? That's the respond part of this. We ship ideas in the market. We sense how it impacts customer behavior. And then based on that learning, we have the opportunity to respond and to move things forward. And our goal as an organization these days is to get through this feedback loop as quickly as possible, because the faster that we get through this loop, the faster we learn.
The faster we learn, the faster we can make the product or the service better, the faster we can optimize it to improve the behavior of our customers. And also, the less we invest in bad ideas. Right? So the faster we can get through this feedback loop, the less we invest in an idea.
And if it does not impact customer behavior in a positive way, we can roll that back much more quickly, spend less time and effort on it. It hurts less to be wrong with this kind of feedback loop. And so this fundamentally changes how we think about delivering value to our customers.
First and foremost, we have to stop thinking of ourselves as a software factory. Right? Because ultimately, creating more code, right, does not necessarily mean that we are creating more value. It just means that we're creating more code that you have to maintain forever in these systems.
Right? And so we have to start to change our measure of success and what that looks like. Because ultimately, the thing to take away from this is just because we can build stuff doesn't mean that we actually have to build those things. Right?
There's probably a way to deliver value much more quickly than overengineering things. And what this boils down to is this concept of outcomes. Instead of measuring output, instead of measuring how much stuff we can make in a particular time, we measure the impact that it has on the behavior of the people that we are serving.
That's what an outcome is. It's a measurable change in the behavior of a human, a customer, a user, an employee, whoever it is that you're targeting with your product or service. And it's these outcomes, these changes in behavior, that tell us when we've delivered customer value.
Because if we see behavior change in a positive way, we know we're delivering something of value. And if the behavior changes in a way that we did not expect, then we have to find out why. Because ultimately, it's these outcomes that tell us when we're done.
And I've got a few stories about this, and then we'll move on to kind of how this kind of morphs beyond just changing customer behavior. But one of the really interesting stories that I've been telling for a little while is this one, and it's about Gibson Guitars.
Gibson Guitars is out of business. They're bankrupt, which is really sad, actually. What happened there is that they noticed that guitar sales were plummeting, and they brought in a new CEO. And that CEO was focused on innovation. Innovation for innovation's sake only. And he thought that by making more stuff, fancier stuff, shinier stuff, innovative stuff, people were going to start buying guitars again.
And he wasn't paying attention to the actual changing needs of the customer in their marketplace. And in fact, if you go to the Gibson website, right, the face of the Gibson website is still the guitar god, right, slash in this particular case, but it's that guitar god.
Conversely, Fender Guitars is thriving. Arguably, and without getting into a religious war about guitars, they sell the same product. Right? Why is Fender thriving? Because Fender paid attention to the marketplace, and they noticed that over fifty percent of new guitar buyers in the marketplace were women.
And they decided to change the services and the products that they offered to these new guitar buyers. This is the Fender website. Right? It's focused on the customer. It's focused on changing their behavior in a positive way, to sell them instruments they want to buy, to teach them how to play those instruments, to provide services that make them better players.
And it's that which is what this particular target audience is looking for. And they can see that by the change in behavior, by the outcomes, by people buying Fender products and services while Gibson is out of business. Right? Now here's the challenge with managing the outcomes.
The challenge in managing the outcomes is that we cannot predict behavior. It's difficult. Humans are complex. For example, let's say you were in charge of making this a safer intersection. This is in Massachusetts, by the way. Right? And you needed to get people to stop at the intersection, just come to a complete stop, and then not turn left.
What would you do? Well, the obvious thing to do would be to put up a stop sign and a no left turn sign. Right? Those features are work as designed. Right? They look the part. They say the right things. They enforce the law in some sense.
But the behavior that we're looking for isn't there. The output that we create is the sign. The outcome is the actual fact that people don't stop at the stop sign. This is Massachusetts. They have a special way of driving there. If you've ever been there, we have a we have a name for those drivers.
Anybody know it? Massholes. Massholes. That's exactly right. That's exact I used to live there. Can say that, so it's cool. So we talked about so we can't predict behavior. It's difficult to predict software systems as well. Right? That's the whole point of these short cycles.
We learn what's challenging, and we move forward. They're complex and unpredictable. The same way that humans are complex and unpredictable and stubborn. Right? We may provide them with a thing that we think that they will use, but it is their preferences that ultimately overrule the features that we put in front of them.
And as we put these things in front of them, we can see their behavior evolve over time. And we can see their behaviors, these outcomes, emerge through the use of the systems that we build, things that we never predicted would happen. Let me share with you a story. This is Instagram.
I'm sure you know it. Right? We have a billion users. Instagram is the place where you go and you post that one perfect picture of the day. So I've walked around Edinburgh. I've taken a lot of photos. In the evening, I find the one that I like.
I edit it, and I post it up to Instagram. That's a lot of pressure, right, to find that right one, to edit it, to get it right. It's lot of pressure for me, a middle aged guy. It's a ton of pressure for teenage girls, as it turns out.
Right? So a teenage girl goes to the beach, someone takes her photo, that part gets cropped and ends up on Instagram. That guy does not end up on Instagram. Right? It's a ton of pressure. Now Instagram didn't realize this when they first built the service, but what they saw was interesting behavior begin to emerge from the use of the system.
And the behavior was this, Finsta. How many have heard of Finsta? A few folks. If you have a teenage daughter or if you know a teenage girl, ask her. I guarantee you she has a Finsta account or several of them. Finsta stands I have two teenage daughters.
That's how I know about this, by the way. Finsta stands for fake Instagram. They're private Instagram groups closed off where primarily teenage girls can do stuff like this. They can take silly photos. They can write on them. They can they can release that pressure of publicly posting that one perfect picture of the day.
Right? Instagram watches this behavior emerge, this outcome, through the use of their system. They're watching this behavior emerge outside of their system with the rise of Snapchat and Snapchat Stories. They ultimately bring this in house with Instagram Stories and continue to evolve this to a point where arguably Instagram Stories are now even more successful than Snapchat itself.
But these are things that they never predicted in the use of their systems. They watched this emerge and they evolved their product to accommodate this new behavior. And we can use this ability to learn and to observe continuously to make our users more successful.
A really interesting story that I came across not too long ago was CityMapper. CityMapper is a is a directions app based out of London for cities, for anything other than driving, basically. Right? And what's the outcome that they're trying to achieve? They want people to arrive at their destinations faster, more consistently, I assume.
Right? That's their that's their goal. Now through the use of their system, they're observing particular behaviors. And what's interesting is that they identified a gap, specifically in London, where people were trying to get from one point to another on a consistent basis, and there was a gap in the transport for London system.
That wasn't being served. That path wasn't being served. This created an opportunity for CityMapper to experiment with actually moving into an adjacent business, and they created their own buses, right, to fill that gap because they saw that there was an outcome that wasn't being met by the current systems, and they thought they could make it better.
And they designed a really nice product around it to match their interface. They even created the interface for the bus drivers to test if this was an adjacent business that they wanted to get into. Right? And so by watching these behaviors emerge through the use of our systems, we can ultimately find ways to make our customers' lives better.
But sometimes, these optimizations that we make can seem trivial, but what ends up happening is they end up creating these unintended consequences, these real world negative unintended consequences. So we had we had Christmasina on stage earlier today, the inventor of the hashtag. Right? Invented by the users of Twitter.
Twitter didn't come up with the hashtag. Christmas Cena did. Right? And then we all adopted it, and we all use it to drive that forward. And really, all of the things, right, that have really made Twitter, you know, an interactive system have emerged from the use of that system, things they didn't predict replies, retweets, those types of things.
And Twitter has monetized those things. In fact, you can advertise against hashtags and all of those things. And we've built the kinds of engagement by evolving the system, the Twitter system, that allows them to see number of users, number of tweets, number of tweets per day move forward.
And it's those exact things that have started to get them into trouble. Right? We're starting to see the heads of these social social media networks get called up in front of Congress and other governing bodies to to explain how they are having a specific impact that perhaps they didn't predict on society.
And specifically, we're then seeing these these CEOs react to the critique that they're getting against that. This is Jack. He's the founder of Twitter. And he says, look. We built Twitter to create this open, healthy, engaging conversation, but we didn't fully predict or understand the real world negative consequences that it could have.
So it seemed trivial. We just built a system for everybody to connect, and now we've got these trolls and armies of Nazis chasing people off the Internet and death threats and that type of thing. And instead of building a systemic framework to deal with this, we're trying to figure out how to make things actually better.
Instead of dealing with the causes, they're dealing with the symptoms right now because, again, we didn't predict this. And and in true Twitter fashion, Twitter gave him the kind of response that he deserved for that comment. Right? But again, their goal is to optimize for engagement. Right?
To get people together to talk about whatever it is they want to actually talk about. And the thing that I want to really mention is that is this. Blindly optimizing for these metrics, the things that we get rewarded for, the things that we incentivize our teams for, without considering these real world implications, these negative potentially negative real world consequences is at best risky, and at worst, it's criminal.
And I wanna show you some examples of situations where teams of product designers and developers and product managers like yourselves were tasked with achieving specific outcomes, and then what actually happened in the real world. Because I think it's absolutely vital to pay attention to it.
So in the name of engagement, right, perhaps the the eight hundred pound gorilla in the room is our friend Facebook. Right? And Facebook has gifted us with the two metrics that many organizations now manage to, the outcomes of of MAU and DAU, monthly active users and daily active users.
Everything at Facebook is focused on getting users engaged on a daily basis and to continue to engage on an ongoing basis, not just on a monthly basis, but on an active daily basis. But it turns out that there are certain places in the world where Facebook is the Internet.
In the same way that AOL was the Internet for a lot of people twenty and twenty five years ago, Facebook is the Internet, in this particular case in in Myanmar. Right? And the platform that was driving the engagement and encouraging people to come back was actually creating a a platform for the proliferation of human rights abuses.
Right? As people were being killed, the Rohingya minority community in Myanmar was being killed, And even the UN is thinking that this may amount to genocide, that Facebook helped with it by encouraging people to come back to Facebook and to engage with it and to kind of rat out their neighbors and talk about where people are and move you know, let people know where these minority folks are.
This is all in the name of engagement. YouTube is the same thing. It also optimizes for engagement. Time on-site, number of videos viewed. And if you watch one video, the algorithms that engineers built, that product managers helped build, right, encourage you to watch more videos like that.
And it creates a platform for the radicalization, in this particular case, of young white males around the world. And what's interesting is as you read about this kind of software creation and production, we talk about optimizing for these specific metrics, these engagement metrics.
Right? But the thing that we don't talk about is, and that no algorithm will tell you, as it says in this quote, is whether that metric is worth optimizing, and what are the real world negative consequences that it actually triggers. Sales. Big driver for a lot of us.
We want to drive more sales. It can cause relationship stress. There was a story about how Target, the American department store, revealed to the father of a teenage girl that she was pregnant before she told him. Because there's a certain type of behavior that Target has identified in their stores and in their shopping that generally means that a woman is pregnant.
And they started sending her coupons and advertisements and ads for pregnancy things. And her dad found it first, and he said, why am I getting this? He went to the store. He's like, are you sending this to my daughter? She's a teenager. She's in high school.
It turns out that she was pregnant. She hadn't told him yet. Right? In the name of efficiency, in making our lives easier, faster, our jobs better, more successful, it can discriminate. Amazon wrote an AI to help weed through the CVs it was getting for job applications.
They they get a lot of job a lot of job applicants. How do we get through that? Well, the biases of the people who wrote that AI showed up in how it was selecting which CVs to proceed and which CVs to downgrade. And it would downgrade CVs with words like women's, women's chess club, the names of all women's colleges.
And it took a while for them to notice this, but the biases of the engineers who wrote that algorithm showed up and began to discriminate out of there. In the name of personalization, these technologically driven products can end up spying on your kids.
We want to create these personalized experiences that we feel connected to. Well, turns out that and this is just one of many stories. In fact, I just read one earlier today about Amazon's Ring doorbell providing video services to the police departments in certain towns.
But this is the smart Barbie. Right? You give this to your kid, and your kid tells this thing anything and everything. Its wants, its dreams, its fears, its desires. Right? And and then they record that, and they keep those recordings with the toy company.
Who's listening to that? Right? What's the child sharing? And can the can the doll be used to help the child? Right? And should it be used to help the child? Right? This becomes really interesting. In the name of convenience, the technologically driven products that we work on can be used to hurt people.
We talk a lot about home automation. Right? We talk about lighting systems. We talk about air conditioning systems. We talk about the locks on our doors and things like that. It turns out that there are edge cases where people are using these to literally abuse their x's.
They turn the stereo, the sonos, way up. They lock the doors. They turn the air conditioning way down because they've got control over this house. But we don't design to accommodate for those edge cases because we don't think about those particular edge cases, those negative real world consequences.
Perhaps in the worst of cases, in the name of news and in the name of consumption and advertising dollars, it can really elevate the worst in all of us, and really bring to power the kinds of folks that maybe we don't want to be there.
And the reality is this. It's not just social media, and it's not just retail. Most industries are using software to change the behavior of their users in a way that benefits the company, not necessarily the user itself. When I was on this stage last year I'm sorry, two years ago, I told the story of John Deere.
John Deere is the Harley Davidson of farm equipment, a really interesting company who's building integrated hardware and software to help farmers become more efficient. And remember, farmers absolutely love their John Deere tractors. Right? This is something that you cannot take away from these folks.
It's really truly amazing. Don't scroll down any further. Trust me. Look, stuff that they're building here is truly amazing. We're talking about self driving tractors, operating two machines at once, the kinds of efficiency that farmers need to make their profit margins more effective.
But there's one catch to all of this, and it's the licensing agreement. You buy a million dollar tractor, but you buy a license for the software. Unfortunately, you can't do anything for the tractor. You can't fix it when there's a problem without accessing the software, and you're not allowed to access the software.
You have to find a John Deere dealer to come in and help you do that. And if you can do that in time, terrific. But if you live five hundred miles away from a John Deer dealer, you may lose your crops in the field.
And the kinds of behaviors that we're seeing, the outcomes that John Deer is seeing, is that you've got American farmers reaching out to Ukrainian hackers to download and buy devices that allow them to crack the software on their tractors and to allow them to access that software in a way that lets them do the things that they need to do.
And when we build this kind of software, we're enforcing corporate policies. And when we enforce these bad policies, people end up working around those policies. That's ultimately what we're seeing. And so as you start to optimize for the behavior of your customers for these outcomes, the things that I want you to think about are this.
Are you solving user needs, or are you exploiting user needs? Both of these things will make you money. The question is, how far do you take it? Now what's fascinating is that this is Ellen Powell in Wired, and she talks about the incentives that the CEOs of a lot of the companies that we like to to to talk about and that I've talked about here today have.
And the reality is that the incentives that they have is reach and engagement. Right? Keeping costs down and driving forward the kinds of engagements that I showed you earlier, regardless of those pesky real world consequences. Because solving for that will inevitably hurt those engagement metrics.
And so as you think about the work that you're doing, and as you think about the folks that you're working with and your opportunities to impact this, Because not all of us are in leadership positions. Right? Some of us just make the software on a daily basis.
A couple of phrases to keep to keep in mind as you come across. I hear this one all the time. I work with engineers. I work with product managers. I work with designers. And they'll say things like, I trust my product manager. They've done the work. They've done the research.
They know that this is a good thing for us to be working on. It makes sense, so I'm going to do it. But you have control over this as well. You might ask more questions. You might be more curious about the impact that this has more broadly in the program of software services that your company provides, and to the customers in the outside world that consume it.
I hear this one a ton from engineers all the time. Look, don't bother me with this real world stuff. Don't bother me with customers and users. I just want to write code. Right? Cool. I get that. Right? That's what's that's your passion. It's what you want to do.
I want to write code. But the reality, like I showed you in that in that Amazon CV AI, is that your ethical fingerprints, good, bad, or otherwise, are going to show up in that code. And you need to make sure that those ethics match the ethics of your organization and of the customers that you're ultimately serving.
And always, always ask the question, what will your code be used to do? What will your designs be used to do? You have to take an interest in the strategic implications and uses of the products and services that you develop. Because guess what?
If you think it's the CEOs that are going to jail, it's not. This is James Liang. He's a software engineer. He's in prison. He wrote the code at Volkswagen that cheated the diesel emission system. Right? What did he think VW was going do with that code?
Right? And he's in jail, not the CEO of VW. Right? Keep that in mind as you move forward. I love this phrase. To kind of bring this home, a couple things for you to think about to wrap this up, things that you can do.
I I learned this from Kim Goodwin, who's a brilliant designer and design leader and manager. She says, look. As we build our digital products and services, let's be goal driven but values guided and measure both. Goal driven are the outcomes that we wanna we wanna change, the customer behaviors that we wanna see in the world.
Values guided are the things that we will not compromise to get there. I love that. Right? So let's go for the sales, the engagement, right, the efficiencies, those kinds of customer behaviors, user behaviors that we're looking for, but let's be very explicit about what it is that we will not compromise in order for us to get there, and raise our hands when we bump up against those those guardrails and those frameworks.
So as you think about what you can do and what you can take away, hopefully, from this talk, a couple things that I hope will inspire you. First and foremost, if you don't already understand your business model, how does your company really make money?
Super important. Take an interest in your customers. If you don't do this already, get to know your customers. Spend time with them, understand their problems, understand their needs, understand how they're using your products to solve those needs. Use the software that you're building to learn so that you can see how well you're meeting those customer needs and how that's impacting your business model and how that business model is evolving based on what you're doing.
Measure the impact of your work. Check those outcomes. See how it's working. See how it's driving. And and don't do it in isolation because inevitably, things have impacts on other outcomes across your organization and your target audience. Alright? Make sure that you understand the relationships across those various outcomes.
And as you start to build these products and services, always bring up the dangerous edge cases. How are we going to protect against these edge cases? They're not going to be the primary use case. They're not going to be the happy path. That's okay.
Right? Discuss the edge cases. And then ultimately, and all of us are in this position today, is refuse the work that harms people. Right? It's in your power. Say, no. I'm not going to work on this. Right? I'm going to walk out. And we've seen this.
People motivate, and they motivate, and they organize. This is twenty two thousand Google employees that walked out to protest how Google was handling sexual harassment at work. Right? They planned it. They made an impact. They communicated, and not a single one of them was fired.
They actually made a big walkout. They made a big deal out of this, and they organized to move forward. And so as you start to build these products and services, Jurassic Park always has great quotes for pretty much everything. Remember this. Right? It's it's easy to focus on whether we can build something all the time.
I'm gonna ask you to think about whether we should. Thanks so much for listening.
