At its core, software is about people: how we interact, identify, and connect. Our relationship with it is personal, but the data defining that relationship is just... out there, often used against us by the technology which claims to solve our problems. Software is not neutral, because it is a reflection of the people creating it, often amplifying bias, prohibiting interaction, and violating privacy - intentionally or otherwise. Consider this talk a primer on design ethics, what happens when we fail to have them, and the need for accountability and solid guidance when it comes to what we build.
← All talks 
Build Engineering Turing Fest 2018
Ethics, Software and Identity in the Age of Data
Auto-generated transcript - may contain errors. Tap a timestamp to jump the video.
Alright. This is working, hope. Good. Alright. So thank you so much for having me. Just a really quick kind of intro to who I am. I work remotely for in Vancouver in Canada, but I was born in Bosnia, lived in Croatia, and lived in Edinburgh for a little while, Glasgow for a little while.
I am kind of fresh off of a plane from Poland right now as well, so any delirium can be blamed on travel and Lufthansa mainly. So yeah. Bear with me. But anyway, before I realized that UX was the thing that I wanted to do with myself, I was a rock climbing instructor, video game tester, and I studied psychology and neuroscience.
And because I I care about what makes people tick and what makes us brilliant, but also what makes us really messy. Messy beings. Our work allows us to figure that out, why we're messy, and to help people on a massive scale. Right? But it also has its problems and that's what I'm here to talk to you about a little bit.
I'm here to talk about ethics. And first, I'd like to get to kind of a common ground about what I mean by that. And that's in in research, so in the world of academia where I sort of got my start and I'm trying to use my degree occasionally, you like before you design an experiment or as you're designing an experiment, you have to ensure that you're doing it in an ethical way.
And by that, mean, you have to have a set of principles behind it that ensure the well-being and the confidentiality of the people taking part, getting their informed consent, avoiding deception, and providing them always the right to withdraw from that. And all of this helps to keep researchers accountable and ethical again, and to protect them from themselves and I think that's a key part of this.
The reason that we need these rules and people to enforce them is because when our careers and our lives in cases depend upon the results of experiments or on the software that we're using, we can't be objective when it comes to evaluating that.
And so here the same is obviously true for software. Those taking part in this case are the people using your service and offering you their data as a result. And you, or your team, the creators of that service, are someone whose livelihood depends upon people continuing to use that service.
And because of that, you'd be surprised just how kind of far back you're willing to bend to convince yourself that the thing you are doing, thing you are making is right. Especially in the absence of any rules to keep you accountable. And so and I don't just mean like design in like a traditional way like how the thing looks and and how it's interacted with, but how how it's built from the ground up.
Like, which frameworks you're using, how your data is structured, how the user experience and the interface work together, and how you market it, everything. And I think that sometimes people confuse this with morals. Like I do sometimes, like right or wrong. And I'm not really talking about that.
I'm talking about the importance of creating rules. Rules for how we act when we approach problems. For, you know, which behaviors are acceptable when approaching those problems and which are not. It's like a code of conduct at a at a conference. It's not enough to say anymore, don't be a jerk.
Right? It's it isn't. We need rules because when we have when we're given any leeway as human beings, we're gonna take it. And I'm also not talking about a one size fits all solution. I think was it Scott who said earlier, innovation is local, I think.
Yeah. So just like in academia, each area has its own version or its own set of rules. And so why not? Why don't we? Like, not as a tech industry, but even within your own business. So today I'm just gonna talk a little bit about why this is now growing in importance, offer some examples, and maybe give a few action items and spark some interesting conversations later, hopefully.
So why it's important is because the scale of our work now is enormous. The sheer scale of data that we store about people and their use of our products is insane. And the more you have to consider when we do that. Right? Like, you have to consider culture.
You have to consider political context of the person who's using their device, their language. And it's also more likely, again, the more people that use your product that they'll use it in a way that you didn't anticipate. Or that action will be taken on their behalf in a way that you or your team didn't anticipate.
The broader you reach, the more people use your product, the more likely you are to harm them, unfortunately. And this is the case because the code that underpins it and the people who underpin it aren't neutral. Right? We are fallible and our output is fallible too.
And in our effort, I think, to solve problems and add technology into the world, we create even more problems in a lot of cases. Like, we're still grappling as I'm sure you've gathered in the news with how to treat each other as human beings, with how to be human to each other.
And yet, we're trying to make computers be human, which I've always found a bit strange. Like, we are the ones teaching those computers, giving them the data to learn. And if we can't bother to define or write down what good behavior is, that's problematic too.
When we're inherently biased, right, even when we don't mean to be, we make decisions on gut feeling and that's brought us to where we are. And oops. Okay. Yeah. And also just because the thing that you like the code that you wrote is good and it works, that doesn't mean that it's infallible or unbiased.
I think there's a lot of nuance here between what we say is important to us as an industry and how we're actually how we actually act. Right? So in theory, we wanna solve problems. Right? But again, there's a nuance here that we seem to be in a lot of ways obsessed with solving the problem as individuals.
It's not like we don't approach it always as this problem needs to be solved, but often, I want to solve this problem or we, like small group of people, want to solve this problem. And this matters for ethics because ethics are about protecting other people but also protecting you from yourself.
Okay. Yeah. And we see this one on like every careers page. Right? But is it I'm not sure how true it is because quite often it seems as though we love to create problems for solutions that we've kind of already have. And this is the equivalent of a researcher already knowing their conclusion and then designing the experiment that'll get there.
And this too, it's not always something to be proud of. It's a it's a good thing in a in a in a vacuum in certain cases. But you'll see why in a second this isn't always a great thing. And before I get to some examples, I wanted to kind of lead with this.
This is from a book called Weapons of Math Destruction that you should read if you get the chance. And remember, again, that the algorithms that perform these processes and the interfaces which take data as their input, they're all made by people. People with opinions and with biases.
So let's Okay. Yeah. And I know like in our industry, we love a good fail sometimes, but I think they're good to to to learn from. So ethics failures can take a lot of forms. Simple as the sale of data, biased algorithms. And all of these usually result from a lack of perspective diversity or self awareness amongst decision makers.
And again, given the scale of these data structures, it's no. Not maybe not data structures, but given the scale of the project, it's easy to pass the buck in a lot of areas. And we shouldn't do that. Try not to do that. And keep in mind that intention, once the harm has been done, isn't relevant.
And so I think again, let's let's start looking at these now. So this was a study done where these various voice assistants were given a variety of inputs. They were all on the theme of depression, abuse or assault. Amongst the things that they did recognize were I'm having a heart attack, my head hurts and my foot hurts.
In which case, they offered some sort of helpful advice to our helpline or whatever. They did not recognize things like I am being abused or yeah. I have a link at the end of the slides to the study if you'd like to to read it.
It's pretty harrowing. I really hate to kick Facebook while they're down and I apologize there's any any from the room. But here, in this case, there's an example of a name not being recognized because it violates Facebook's name standards and this is an indigenous person.
Someone someone programmed that. We created that. And think about it. If you've got a bunch of people in the room who can who can write code or who can design, what are the chances that they reflect the people they are meant to be helping in terms of age, race, gender, sexual, everything?
So when you're deciding on a default, because there always needs to be a default. There needs to be something, but look around to see who is in the room and if that matches the people for whom the problem is being solved. Here's another example.
Nextdoor, for those who don't know, it's a social network for the neighborhood. It lets people make posts about their area, good or bad, like about events and and things, but also like, oh, there's something weird happening in the neighborhood you should know. In twenty fifteen, it was discovered that the platform actually allowed people and enabled people to racially profile and it became a big problem.
It was filled with posts of suspicious activity which had which often had little in common other than they were people of color trying to go about their day. So here's an example where, like, no crime was committed. It was just a young black woman knocking on the door and asking for Keith, and then she left and that was it.
And this person felt the need to warn the entire neighborhood about it. Maybe some local folks have heard of this, Pure Gem, a couple years ago. A woman named Louise Selby in Cambridge tried to swipe into a gym, she couldn't get in. It was because she'd signed up as doctor and she couldn't get into the female change room because doctor was male.
So The software that managed the that managed their membership data, coded doctor as male. Orbitz and Wells Fargo, I think yeah, a couple of years ago as well. Wells Fargo especially, a hundred and seventy five million dollar settlement for discrimination because its website varied prices and steered about thirty thousand people of color borrowers in the early two thousands into more costly loans.
Because it yeah. And Orbitz steered users of Apple products to pricier hotels than others. This, like, just happened a couple days ago. According to a test conducted by the American Celebrities Union, the Amazon's facial recognition AI matched twenty eight members of congress to arrest mugshots.
And these false matches disproportionately affected members of the congressional black caucus. Yeah. And furthermore, also recently Facebook, Instagram, and Twitter provided data access on their platforms to a surveillance product that was being marketed to target activists of color as well. And here's someone showed this, I believe yesterday?
Or earlier today? Yeah. Sorry. So this is a soap dispenser in a in a bathroom. Just give it a moment. Onto your honey. Too black or too black? Yeah. Yeah. If you can hear me sing too black. Come again, Sasha. Hello. No. No. Wait for it.
There you go. So It's not ideal. So you can see that all of these are different different forms of ethics failures because oversights in the building of of these pieces of technology resulted in people being harmed in one way or another. And all of this was avoidable and I've imagined not intentional.
Right? But it happened anyway. And so a casualty of this is identity and I I wanna make a special point on this. It's that algorithms make a series of assumptions about who you are. And all of these assumptions come from those who create them and our individual identity suffer as a result.
Here's another quick example. So this person got a push notification to from Etsy to get a gift for him for Valentine's Day. And but what if your partner isn't male? And find an isolated incident like this, you know, could be funny. Like, I'm gay.
And if I got a push notification like this, I would be like, oh, that's funny. But imagine how this compounds. Right? When every the majority of products you use in some subtle way or another don't reflect who you are or don't take into account who you are.
And here's another one. OkCupid asks how you identify. You can select up to five of these but then really, as the person says, okay. But actually though, are you a man or a woman? Like, where do you wanna be included? Which is also kind of misses the point.
Yeah. Transparency and trust again also are casualties of this because this is one of the underpinnings of ethics at least in the field which I'm familiar. Transparency. Not to literally everyone but to to the people who are building and using your product. So they have a right to know not just what you're doing, but they must understand it before giving consent.
And in this case, I'm looking at the privacy policies that are five hundred pages long and whatever. But so many of these things are algorithms, privacy policies, whatever are black box and nobody knows what they do and therefore they can't be held accountable by you or me.
Another casualty self expression. People may avoid expressing themselves online or participating in any sort of public debate or refrain from doing anything that might be considered unwanted unconforming. And such things become especially problematic or difficult when extensive information about your everyday life determines access to things like financial services or employment and other vital opportunities.
So this is an example of that really really interesting notion that when someone else somewhere literally owns who you are, like, we should be regulating that. We should be aware of that. And finally, autonomy. Because again, this changes behavior and creates stress and fear and harm.
You might not visit certain websites or use certain products, interact with certain pieces of content, whatever. Or searching even certain terms because you're afraid of what might happen to you, how that might be used against you. You know, this means that you adapt your life to technology and not the other way around.
Because, like, if I think about, like, the weird crap I did online when I was a teenager, like, think about like when you like, the internet was like when you first encountered the internet and the kind of stuff you search for, the kind of communities you'd be a part of.
Would you do the same thing now? Would you partake in that same thing now? And with the same sort of at least for me, with the same sort of zeal and excitement. I'm not sure if I would. So I'm now going to attempt to give you some sort of practical advice.
And this is gonna cover a range of situations, like if you've already got a product, cool. If you're thinking of building one, and so on. And this is very much again at the local level, especially if you're in a position of privilege at your company.
You can define your own ethical standards. We don't need to wait for some sort of, as in academia, like independent board or whatever. So yeah. We should be talking about these things. I'm just gonna try and give you a few places to start.
And I first do want to address this particular elephant in the room, which is GDPR compliant now that we've all stopped receiving all of the emails. For the most part anyway. I I do want to address it and it's just a host of changes for how you store data, for how long, and the access that folks get to it.
It protects individuals' rights to privacy. And I just wanted to underline that and just say we should comply with it. And beyond that, this is the most important piece one of the most important pieces of advice that I'm going to try and give.
If you're thinking of building a product or if you already have one, we need to ask ourselves some hard questions, like actual like introspective questions. Because and I'm guilty of this. We want to jump headlong into solving the problem before stop like really stopping to think because we want to help.
Right? But this is a key question. It's and examining our motivations is hard because we might not like what we see or the answers that we give ourselves. Because okay. What problem does your product solve and for who? Like, really for who? And why do you or the people around you want to solve that problem?
What is in it for you? There's always something in it for you. Yeah. And then the next question comes in is should you be the one to solve this problem? Like, And a valuable example here is iOS HealthKit a few years ago, I think now.
When it first shipped, it didn't have reproductive health tracking for over a year despite billing itself as the app that you could monitor all of the things about you that you're most interested in or whatever. Think it was The Verge who asked in an article reviewing it saying, Is it really too much to ask that Apple treat women and their health with as much care as they've treated humanity's selenium intake or sodium intake.
Because you could track both of those things but you couldn't track your reproductive health. And Fitbit, very recently actually, it was found that they wouldn't allow women to track periods longer than ten days which FYI, that does happen. And, like, when you try to when you try to input the the date range, said, nope.
Maximum is ten days. You can't do that. So and this was the executive team for the for iOS when the reproductive health problem happened. And again, fine. But like, these people need to be involved somewhere. The the affected folks need to be involved somewhere along the decision making process.
They also another example, they forget people whose primary interest in these apps isn't a baby necessarily. It's not about pregnancy, it's about other needs. And that's not exactly inclusive either. So now that we've moved along that, ask yourself also what data do you need?
Which kinds of personal information is used? What needs to be what needs to be used as input? And, you know, how will the person volunteer it? How will you acquire it if you do? And then, as I've kind of threaded throughout, build the right teams.
And by this, I mean diverse teams, inclusive teams. Do they look like the people you're solving for? And we don't really have an excuse for this anymore. There have been some great talks throughout that have touched on this, so I I won't stick around too long.
Inclusive interfaces are built by diverse teams. And actually, like, achieve the theoretical tech dream. Right? That making the world better for everyone. And yeah. And Maria earlier today mentioned that all of this can be helped by asking the right question at the right time.
And that is about making sure the person who asks that question is in the room. And so you can hire these people or you can hire external consultants as well. Because these people will tell you what you don't want to hear, and that is important.
And yeah. I just wanted to reinforce this. So what's next? The regret test is an interesting thing that you can also leverage. If the people using your service knew everything, well here it's like the product designer, but okay? Or the engineer or whatever.
Would they still execute the intended behavior? Are they will they regret what you've just made them do? You can plan for misuse cases. This is you you can ask external expertise with this as well. We often ask people to audit the security of our of our systems.
Right? So why not ask them also to test whether or not their product can be used unethically or will result in unethical action being taken for the person. This is also similar to stress cases too. So not average use cases, but when might someone use your product in a stressful situation.
NPR used this to, I think, fairly good effect. And this is an example of a stress case. A human might be anxious about a breaking news event, worried because it personally affects them or their community, and is afraid of receiving inaccurate information. So you put yourself in the mindset of someone who is encountering your product at a stressful time.
Yeah. So this is a quote from one of the team members. It says, these cases help you design for real user journeys that fall outside of the ideal. And as a designer, this is kind of maybe a problem, but you can foster empathy with your user in a lot of ways.
You can do research. You can interview people. You can shadow them. You can give your engineers access to customers to interview so that they really understand them. But in my opinion, there's only so much empathy you can build. Okay? No amount of design thinking, quote unquote, is going to help me understand, for example, what it's like to be black or to experience the world as a person of color.
I and therefore, I should not be a decision maker on a product meant to serve those people. I can be involved perhaps or engineers can be involved, but only in that operational sense. The final decision should be made by those affected particularly when it seems to run counter to what you think.
So I argue instead for humility. Know the answer to the earlier question, when are you not the right person for this? But again, you can still help. You can use your privilege in other ways. Another thing. Measure what matters to to to these people.
Just because this is a common example, but just because people are spending time on your platform doesn't mean they're getting what they need. For example, in my job, we're a platform which allows people to send targeted messaging to their customers. So emails, push notifications and whatnot.
But for us, this situation might not mean that our customers send lots of messages for to their end user. It might mean that they send less of them, actually, because but they're just more helpful. Ethical design doesn't necessarily mean that they spend less time with or it might mean that they spend less time with you, but you retain these people for longer.
Transparency, again, important in levels. If you need to use data, for example, be transparent and be human and tell people that you're using this. That's fine. Know the answers to the earlier questions. Where is it coming from? What are you collecting? Talk about it. Be transparent. Again, regret test.
If you were told this, would you continue to use this product? I think it was I saw Norwegian dot com had good booking system where they they told you exactly like it was all their fees were out there upfront. And then document these.
This also again, ask yourself all those questions, do the introspection, have some self awareness, but then document them. Because without documenting these and making them public, there is no way to hold ourselves accountable for it. Right? This takes time but it's it's worth it because and though it is sometimes difficult to quantify, the value of all this is the absence of harm.
That's why it's difficult to quantify because a thing that doesn't happen is important. But it's difficult to quantify, but that doesn't mean that we shouldn't try anyway. So again, like, yeah. Given recent news, I don't think that Mark Zuckerberg is the person that we want to be like getting our guiding principles from anymore.
So, like, no more moving fast and break things, moving slowly, build with purpose. You can you we all say, I'm I'm an avid learner. I'm a voracious learner. Like, all this on on our on our CVs in our in our companies. So let's, like, actually do that.
Again, I'll I'll have these slides up later. That's a link. There's like two hundred courses in tech tech ethics if you wanna take one. Read some books. Read some there's some wonderful organizations doing great great research. And remember that what you say doesn't matter.
And that's why again, it's you must be transparent and write things down. Because that's the only way accountability can happen and you are what you do not what you say. This is yeah. And furthermore, mistakes are fine. Mistakes happen. We're again, we're fallible. It's okay.
What matters is how you fix them. And I'll I'll show an example of the next door. So they looked at their platform and were like, okay, we don't wanna we don't want people racially profiling. So when one change that they made was when race was mentioned, they forced people to add more information to their report.
To make the end what this did was it made them think a little bit more deeply about what they were describing and whether or not it was harmful. And it and it while the platform does continue to experience its problems, this was affected to a degree so it can happen.
You need to just recover. And I thought this was really cool. This was part of the hackathon here. There's a team that built a game and in their presentation, one of their slides was ethics. So it was a game called Earthbound. There was a If the next generation is thinking about this, we should be too.
And I wanna close with this because I think it's important. I think admitting this is actually what matters and not even in a facetious way. No. Like, don't know what we're doing. That's alright. But it's time to acknowledge that and act on it in a certain way, in a in a self aware way to to gain that self awareness because we can use that then to make better decisions or empower others to make the right decisions and help us force these sort of or create these
codes of ethics or principles of ethics within our within our companies. We can know which decisions we should be making and which to leave to other people. This is actually one of those hard problems that we should pride ourselves on solving. It demands all the skills that we claim to have as technologists and and builders of software, like collaboration, innovation, technique, helping people.
We can solve this one sort of in individual companies and then together and actually make people's lives everyone's lives better with things that we make. We can let data drive our decisions but only after the rules that we give that data ensure equality across the board for everybody.
Thank you for listening. And I have I have all these. I'll I'll I'll tweet about the slides once I get a speaker deck working. But if if you wanna read any of that, I'll I'll have it all up for you.