We all know how important it is to keep the user at the centre of product development to achieve PMF and grow. But gathering and effectively leveraging user feedback to shape strategies and roadmaps can often pose a challenge – whether from a resource, skill or tooling perspective. As a result, we risk missed opportunities or unsuccessful product launches by failing to take the user perspective into account. Having run research at every stage of maturity from solopreneur to leading large teams at Facebook, booking.com and Skyscanner, Nyssa will share strategies for conducting user research in a relevant way for any size of business – from startup to big tech and everything in between.
← All talks 
Build Marketing Turing Fest 2022
How To Run User Research At Any Scale – From Startup To Big Tech
Auto-generated transcript - may contain errors. Tap a timestamp to jump the video.
Hello. Hello. Welcome. Thank you for joining me. And as James said, my name is Nissa Packard, and I lead the insights team at Skyscanner, which consists of data science, consumer research, and user research, which we're here to talk about today. How you can get started with user research no matter what size or scale your business is.
So given my role, I naturally love asking questions, collecting data, and sharing insights with people to help them make better decisions to fuel business growth. So that's what you can expect for the next thirty minutes. Strap in. There's gonna be a lot of information we're gonna work through together.
And this is something really near and dear to my heart because as a former founder myself, I'm very passionate about helping small businesses succeed. So I hope that you'll come away with this from this talk with lots of practical tips that you can apply to help fuel your business no matter where you are in the growth curve.
So let's start with question number one. Are you ready? Who knows what user research is? Raise your hand. Pretty good. All right. Anyone care to venture a definition? Just shout it out. Come on. And Tessa, if you're in the audience. Asia, I've seen you guys talk about this. Anyone?
Any guesses? Identifying user needs. The understanding user needs. Thank you, brave soul. Let's give her a round of applause. Yes. That's exactly right. So it's about understanding user needs, motivations, and behaviors. And we do this with a variety of different methodologies. It can be tasks, observations, or other types of feedback.
So that's all well and good, but why should you care? Why are you here to hear about user research? Because ultimately, it's not just about understanding user needs. It's about making sure that your business stays in business. According to an analysis of over three hundred failed startups, it turns out that half of those top twenty reasons why startups fail have to do with the customer.
And the number one reason that they found why startups fail is that they didn't understand the customer need that they were solving for. There wasn't a customer need for their product. So it's critically important that we take the time to make sure we understand what that user need is, what that customer need is, to make sure that we will achieve product market fit.
The Harvard the Harvard Business Review also did an article on why startups fail and and found the same thing and went went further to say that by neglecting to research customer needs before commencing engineering efforts, entrepreneurs end up wasting valuable time and capital on MVPs that are likely to miss their mark.
Likening entrepreneurs in this scenario to sprinters who kind of jumped the gun. They're so anxious to get their product out there that they failed to actually make sure that they're meeting a need in the market. Does this resonate with anyone? Very anxious to get their product out there.
I know when I was a I was a founder, I was very keen to do myself. So it's something that we we're all susceptible to and something I still see at some of the biggest tech companies. So ultimately, user research isn't just about understanding needs and motivations.
It's about saving time, helping you accelerate. So Paul talked this morning about, you know, this is a race. It helps you get faster to market with the right solution so you don't have to test and learn for nine months before you get there.
It helps you save money on engineering time, design time that doesn't need to be spent until you first know what you're solving for. And it ultimately reduces the risk in the decisions that you're making. Again, as Paul said this morning, building product is making a series of decisions.
Research helps you reduce the risk in the decisions you're making so you can make better decisions and ultimately grow your business. So that's what we're here to talk about today. But you don't have to take my word for it. Let's hear from several of the founders of these three hundred failed startups to see what they have to say in retrospect was the reason behind when when things went wrong.
So the founders at eCrowd said, we spent way too much time building it for ourselves and not getting feedback from prospects. It's easy to get tunnel vision. The founders at Treehouse Logic said, startups fail when they're not solving a market problem. We had great tech, great data, great reputation and expertise, great advisers, etcetera.
But what we didn't have was a business model that solved a pain point in a scalable way. And finally, from Voter Tide, the founder said, it's easy to get tricked into thinking that your thing is cool. I think we all can recognize that.
You have to pay attention to your customers and adapt to their needs. And I think that really hits the nail on the head that we often can get into the mindset that we are our end user and we end up building for ourselves.
That is not the case. So that's why you need to go out and talk to actual users or customers. So I've seen this myself from multiple angles. As I said, I've I've been a founder myself. I've been in your shoes. I've been a researcher.
I've I've been researcher number one on the partner side at Booking dot com. And I've built up research teams and and led research teams there at Booking dot com as well as other companies like Meta and now in my current role at Skyscanner.
Any Skyscanner users out there? Yay. I like to see that. So for the five people who didn't raise their hands, we are the world's largest travel marketplace, and we're very proud to have recently won best travel comparison site by Witch and Farmers. And we have a lovely booth here in the expo, so go say hello to my colleagues.
We're recruiting for engineering roles, for data science roles, and and for many other roles. So go say hello to hello to my lovely colleagues. Alright. So at Skyscanner, this is how I frame the mission of the user research team. It's around delivering actionable insights that improve user experience and accelerate business growth.
And we have two key jobs to be done. The first is about landing the current product roadmap as successfully as possible with our users. That's typically what we call evaluative research, which I'll get to in just a minute. The second is about identifying the greatest opportunities for future business growth.
What is that untapped opportunity to meet a need better than others, better than competitors do or in a way that we don't today. So identifying what should be in our future road maps. And that often comes from what we call generative research. So as I said, this is going to be a very practical talk with lots of tips.
I'm going to talk about what types of questions and decisions user research can help inform for your business, how you can get started if you're not doing user research today, and what tools you can use for free. Again, I know I've been there with limited budgets.
You gotta have some cheap options available as well. As well as what to look for in the first user researcher you hire, and how you can build out that team successfully as you scale. So you probably have a product development flow similar to what we have at Skyscanner that starts off with, you know, the discovery and inception phase, goes through concepting and ideation, to building a plan, a minimum testable product, or maybe an MVP, as you might call it, AB testing or experimenting, launching, and monitoring it in the long term.
So where does research fit into this equation? Well, throughout most of the process and especially in the earlier phases. So in the early days of discovery and inception, we do generative research, which might address questions like, how can we achieve product market fit?
How can we simplify our information architecture? How can we improve customer retention? Or where are the future growth opportunities for us in new verticals or new markets? Once we have something to evaluate, whether that's an idea, a concept, a prototype, or even an existing product, we can then move into evaluative research where we can address business questions like, how can we reduce cart abandonment?
How can we improve conversion? Which new feature should we prioritize? Or how can we improve the launch of our new feature to land as successfully as possible? Then through experimentation, this is not often led by user research, but sometimes user research plays a role here in assessing how much did the change we made move the needle.
Or in the monitoring phase, of course, looking at I see lots of phones going up. Snap away. What should we focus on to optimize the user experience moving forward? And this can be through tracking research or through looking at analytics data. And so then we start the cycle again.
It's, of course, a circle. And not all decisions are created equal. So as we said, conducting research is about reducing risk and saving time and money. So ultimately, the more risk that is associated with a decision, if you're changing your check out flow, payment options, redesigning your home page, or if you're doing something that requires a lot of engineering effort, you know, building a new app, these are the types of things that are most critically in need of user research.
On the other hand, we might have some more local maxima that we're optimizing for tactically, like testing different colors of buttons, different text options, different adding different filters. Those are not necessarily things that need research. You can just AB test. But it's the big strategic bets that you want to make sure that you inform from your users.
Alright. Question number two for you all. How many of you are conducting user research currently on a regular basis within your companies? Alright. I see about a third of the hands going up. That's great. Glad to hear it. Carry on. For those of you who didn't raise your hands, this section is for you.
How do you get started if you haven't done user research before, especially if you're budget constrained? So I will preface this by saying this is a profession. It is a lot more complicated than what I can explain in thirty minutes, but I will give you a crash course with the fundamentals.
So you always need to start with your business objective. That sounds very obvious, but I can't tell you the number of times that research projects go wrong because this wasn't clear from the start. So get crystal clear on what you're trying to achieve.
For example, we might want to say, how can we we'd like to improve the number of customers that sign up themselves so that our sales teams don't have to manually go out and sign up customers. We wanna increase sign up of new customers via a sign up flow.
There we go. Second is to identify the research questions. So that'll be based off of your business objective. In this example, that might be, where do people get stuck in the sign up flow and why. And from that clear research question, we can then identify which type of data we need to collect and what is the best way to do so.
So with this example, we clearly would wanna understand in the sign up flow, looking at the analytics data, where do we see the biggest drop offs. And then secondly, we would wanna do some user research, some contextual inquiry or usability testing to have people go through that sign up flow and see why they're getting stuck at those places.
What did they expect to discover? What don't they what information do they not have available? Or what do they misunderstand? So those are the basics. And there are lots of different methods that we can pull from in our toolbox as researchers. And we typically divide these into a couple different categories.
So we've got qualitative and quantitative. Qualitative would be methodologies where we're talking to typically less than ten users per subgroup that we're interested in. So that would be things like interviews and focus groups that you're probably familiar with. That's the type of attitudinal research where we're asking people about what they want, what they need, how they feel about a current experience.
And then we have behavioral methods as well, where we're looking at how people behave. Surprise, surprise. So that would be things like ethnography, where we observe people in their natural environment and see what they do. Or usability testing, where we give them a task, they're engaging with your product, and you observe how they interface with your with your product.
We also have tools like diary studies to look at that longitudinally over a series of weeks. For example, with travel planning, that's often that often spans several weeks. So we would do that type of thing. We have eye tracking to see where people are actually looking on a page, what they're missing, what's drawing their attention.
And tools like card sorting where we can look at what are the mental models people have about how things should be grouped to inform your information architecture or to create bundles for products, for example. Then when we head into the quantitative side of the graph, we've got hundreds or even thousands of responses here from from hundreds and thousands of users.
That includes things like data mining and AB testing, which is in the millions or or more data points. We've got surveys down in the attitudinal side of things where we're asking large numbers of people what they would like, what they prefer, what they need, how good a service was, that type of thing.
And then conjoint is more sophisticated technique where a series of options are given. And based on the choices they make, it extracts the importance of different elements from those choices, different features, different price points, etcetera. This is not comprehensive, but that gives you a bit of a foundation of the types of research methods available.
So how can you get started for free if you want to start doing research as a startup? So these are actually the tips and tools that I applied myself. I'll talk about my case in just a moment. But for things like interviews and focus groups, you can recruit people with Calendly, which is a sign up tool.
You can set the time slots yourself. People can sign up themselves, direct them to Zoom where you can record the session, and boom, you have your session recordings of interviews to analyze after the fact. That can be done for usability testing as well.
If you want to not do remote testing, but you wanna intercept people live and have face to face engagements with your users, you can do what we call gorilla testing. Has nothing to do with primates. It's where you intercept people in likely environment for them to be in.
So if you have a fitness app, you might want to intercept people outside of a gym. And then you can ask them for feedback on whatever topic you need information on. We also can use tools like Miro for card sorting, which you might use in your businesses.
It's a virtual collaboration platform with with virtual post its. So that enables people to group different ideas and features into the most logical mental groupings for them. In terms of behavioral options, of course, we have dogfooding, which a lot of us are familiar with, where you put yourself in the shoes of your user and you go through the user journey yourself.
So at Skyscanner, that would mean we would book a hotel for ourselves. We would book a flight for ourselves. And as we go through the process, we look for where what is confusing? Where is their friction? Where did something happen that I didn't expect?
You make note of those things to help improve your product. Immersions would be observing someone else in their natural environment and and making note. So for example, if if you're at Uber, this could be a drive along with an Uber driver where you sit in the passenger seat and you watch for the day.
What is the experience like of the driver, of the of the passengers, etcetera? Then there's a great tool that I love called Hotjar, which spans the behavioral spectrum from qualitative to quantitative because I hope Hotjar is here. I love Hotjar. They do session recordings of what the user actually does on your site.
It doesn't record any personally identifiable information, so it's all safe, but it's also unbiased behavior. They don't know they're being recorded. So you can see where they actually get stuck, where they're rage clicking, where they can't fill out a form, whatever it might be.
And they also aggregate this data at a quantitative level so that you can see heat maps of where people are clicking, what they're missing. It kind of simulates that eye tracking type data. And also, you know, how far people scroll, that type of thing.
In the in the analytics quadrant, of course, we have things like Google Analytics, Mailchimp, Google Ads, Facebook Ads, a lot of things I'm sure you're all using where you can get your AB testing data on on product changes and on marketing campaigns. And there are a couple tools that you might be less familiar with, like Google Trends and Facebook Audience Insights, which zoom out a level further in terms of data.
It's not data about your product or your marketing. It's data about the market at large. So Google Trends looks at all of the search data on Google. So you can see what types of topics are trending, what the search queries are around that topic, where geographically those searches are coming from.
It's a wealth of of useful data. And Facebook Audience Insights uses all the profile data across Facebook and Instagram to help you get a better understanding of the profile of people who like a given topic or use a certain brand. Then when we get to surveys, you have a couple options.
There are many free survey tools like Google Forms there's one more click if you want to take a photo that enable you to write surveys for free. The challenge is how do you get those surveys distributed to people to answer your surveys? So you can send it out through your social media channels to your followers.
That's totally free. Or if you wanna reach a broader base of people who are of a specific demographic, you can do that as a boosted post or even run it as an ad, which enables you to do the same type of targeting you would for your ad campaigns.
You know, get women eighteen to thirty four who live in a certain region, whatever you like. This is the last click if you want a photo. There are also survey tools like Google surveys or quick surveys where you can connect directly to a consumer panel or a series of users, and then you end up paying per response.
So on Google surveys, for example, I think you start paying around a pound per response for a short survey, which means you need two hundred responses. That's two hundred pounds. So not totally free, but still on the cheap side. So this is there it is.
This is a suite of tools that you can use to get started. I do recommend us looking at some how to guides for how to write a good survey, how to design a discussion guide. Because it's not just a free for all that you use the tools and you'll be good to go.
You need some kind of understanding of how to do it right. Alright. So to give you an example of how I put this into practice with my business, my business objective was how could I build for a product market fit? I had done some wine education studies.
I'd worked in e commerce for a while. I wanted to do a wine e commerce business, but I didn't know what type of wine e commerce business I wanted to do. So I wanted to get to that winning idea. I started with market analysis.
So using Google Trends, I looked for which types of countries had the most growing demand in terms of interest in wine, what types of wine they were interested in. Turns out Prosecco in the UK continues to be increasing popularity every year. And nonalcoholic wine was a big trend.
If anyone's doing nonalcoholic wine, there's a big opportunity out there for you, but I couldn't find any good nonalcoholic wine. So researched the different, you know, the the wine market in general. Then I looked at competitors using Facebook audience insights to see what is the profile of people who are currently buying from brands like Naked Wines.
What are they interested in? Turns out, nesting and travel. So that helps me with my marketing campaign later on to know what types of interest areas I can piggyback on or what kind of co branding I can do. I then went through the critical stage of looking at the jobs to be done, which I think Tessa mentioned earlier this morning as well, and used the value proposition Canvas.
So how does that work? The jobs to be done who's familiar with jobs to be done? All right. There's some level of familiarity. So jobs to be done, for those of you who don't know, is a methodology that is based on what people are trying to accomplish and the theory that they hire different products to meet those needs.
So the job in my case was not about buying wine, drinking wine. It's about enhancing a meal, impressing guests, unwinding, even being perceived as refined. That's what people are trying to do. And they can do that with wine. They could do that with cocktails.
They could do that with a book. There are many different products that can meet the need of that job. From that job, you then look at the pains and gains. What are the things that help people achieve this job? And what are the things that can get in the way and make it difficult?
So for me, one of the things I heard was not knowing about wine and the risk of getting it wrong and and actually ruining a meal or or embarrassing yourself in front of friends. From there, you can go into shaping your value proposition, which means looking at which gains can you enhance or which pains can you reduce.
So for example, having a curated selection of wines done by an expert and having pairing tips for which food to to pair with your wine and even some recipes that you can make that'll go perfectly. So ultimately, that's how I got to my value proposition around a small business focused wine subscription.
From there, if you unearth several different jobs to be done, which is likely, you can size those. Looking at how big are those jobs in the market, how many people are trying to achieve that job, and how satisfied are they with the current options available.
And obviously, we'd be looking for the sweet spot of jobs that are both sizable and are underserved in the market. Because that mean that's low hanging fruit for us to go meet the needs of these people. And from there, we can start to ideate.
So this is the concepting phase where we can test out different ideas and see how appealing they are to our target demographic, as well as how unique they're perceived to be relative to the other competitors that are already out there. So these are some of the concepts that I actually tested for my business.
It's a snapshot from the focus groups I ran at my house. I also did interviews at my house. Like I said, all free. And, you know, tried out a natural positioning around organic, sustainable, a world tour about discovery of different wine regions in different countries.
And the winning one was the hidden gems around, you know, niche boutique wines that were hard to find from small family vineyards. So that was what I moved forward with. So then we've got the trifecta as the end product here. I'm sure you've seen this before.
So we've now assessed the desirability from the research. But then we need to look at the feasibility, which is where you need to talk to your engineering partners and say, can we actually build this? How can we actually build this? I'm looking at our head of engineering right here.
And then the viability, which is about profitability. Can we make money with this? How much are people willing to pay and how much will it cost us? So how much people were willing to pay was another thing that I gathered through research. How much will people pay for a three bottle box, a six bottle box, a twelve bottle box, etcetera.
I also tested things like the branding to look at both what was appealing to people as well as what was actually memorable, which brands could they recall after a five minute survey at the end spontaneously, which meant it would be easier for them to communicate it via word-of-mouth because they could recall that name.
So you can see this is how I ended up with my brand. The most memorable, Vino, as well as well as one of the most liked, The Curious Wine Club, which gave me Vino, The Curious Wine Club. It's a bit nostalgic looking back on this.
This was several years ago. And I can very much commiserate with the amount of stress and uncertainty that a lot of you are dealing with right now. Unfortunately, I had to close down my business because I was diagnosed with cancer. No worries. I'm in remission now for four years.
But, yeah, the stress that you go through is is very palpable. So I'm hoping that these tools will help you to make better informed decisions to help fuel your business growth. So in case you didn't snap the photo the first time, this is once again the suite of tools that you can use to get started.
Now, your business is continuing to grow. You're ready to hire your first researcher. Who should you hire? And how do you get started? So in order to set yourself up to succeed, you need to first hire an experienced user researcher. There might be times where hiring someone straight out of university.
This is not the time. You need someone who knows what they're doing, can hit the ground running, and be autonomous. You also don't need the most experienced person who's led teams because you want someone who's gonna get their hands dirty and and be a bit scrappy.
You also wanna set priorities both for the person coming on board as well as for the team the rest of the team about the fact that you're going to be using research moving forward to help make your decisions throughout product development. And setting the priorities for the researchers, again, about those business objectives.
What are the key things that they should focus on solving for? Because inevitably, what I hear from many researchers who are the first is they get inundated with requests and they don't know where to start. So it becomes about who's shouting loudest or who do they have the best relationship with.
And that's not how you want them to be prioritizing their time. And perhaps the most important piece is about integrating this into your prod dev process. So you saw what our flow looks like at Skyscanner. Think about how you communicate this to your team and your flow about when and where you'll be using research to inform your decisions.
Now, who should you be hiring? This is a great analysis based on seventy different user research job postings from the past year. Thank you, UXPlanet. And you can see what companies are typically looking for when they hire a user researcher. So you can see the top piece that's important is the methods that this person has experience with.
And in terms of methods, the most important methods they really need to know are surveys, usability testing, field field research, and interviews. With those four under their belt, they'll be able able to cover the vast majority of the needs your business might have.
Beyond methods, we then have things that we would expect from most team members. Things like communication, project management, collaboration. And then we get to things like experience and qualifications. And I would say the experience is much more important than the qualifications here. Having had people in my teams with backgrounds like history, foreign languages, English literature, it doesn't really matter so much what they studied.
Human computer interaction is the most common along with psychology probably. But I think having the years of experience is far more important here. So you'll probably wanna target that three to five years range, which you can see is kind of a sweet spot in the market as well.
So that you have someone who's autonomous and able to hit the ground running. Alright. You've got that first person started. There's a lot more momentum. People wanna learn more from your users. So how do you build the team to scale with your business growth?
You're likely to start off with one of two models. This is the one I recommend, which is the agency model, where the team will work on projects like an agency for different product teams across the business. That means that they need to prioritize ruthlessly on the most important things, the most important questions for the business to answer.
This is how we approached it at Skyscanner recently because as you can imagine, coming out of COVID, the team was pretty depleted. So we had only one or two researchers at one point, and we needed to move to operating like an agency model for a while.
And so we took all the research requests we would receive, bundle them into requests that could be handled in one project together, and then prioritize those with the c suite to make sure that we were working on the most important requests for the business.
It seems very simple, but if we hadn't done this and just taken that kind of first in first out approach or prioritizing based on who's shouting loudest, we probably would have only taken on three projects instead of the twelve that we were able to take on by bundling and prioritizing effectively.
The other option you have is a democratized model. This would be having that same small set of user researchers at the core, but using them more as mentors to help PMs and designers across the business to conduct their own research. So there are some advantages of this in terms of you can scale the amount of research you can do very quickly because you're just training up your existing workforce to be able to get started.
But there's intrinsic bias in in the research that'll be done by PMs and designers, not because of any ill will, but because, you know, it's their idea, their design, their baby. No one wants to hear their baby is ugly. So you're intrinsically not gonna be very objective when you're testing something that you have worked on and and put blood, sweat, and tears into.
And secondly, it takes time away from the work that they would otherwise be doing. Time away from building product, from designing, etcetera. So this is it's an option I've seen businesses apply, but it's not my recommended approach. I would go for the agency.
Then as you continue to scale, there are two options there as well. You can try an embedded approach, which I've seen in some businesses where you have user researchers reporting directly to the the head of product for their area. This gives them lots of contextual understanding of the product space and enables them to be proactive in identifying what other research might be needed next.
The downside is that they operate in silos, and so you might get duplication of work across the squads. And you lack that connective tissue to inform strategic insights that don't fit within one part of the product, but are the bigger picture business type insights that you need.
It's also not great from a personal development pro perspective in that people aren't getting that mentorship and exposure to how others work to grow in their career. So this is my preferred model and what we've applied at Skyscanner, which is a hybrid approach where we have user researchers dedicated to each of the product teams so they have that context and they're able to be proactive.
But we also have a core centralized reporting line, which avoids duplication. It enables personal development and taking a holistic strategic view across all of the work that we're doing. We also have champions that we empower across the different product teams to do their own self serve research with templates and how to guides.
And a shout out for research ops as well, which is a key enabler of efficiency, making the best use of your team's time. So I told you it was gonna be a lot. So we've gone through why research matters, how you can get started, what kind of questions can be addressed with research in your business, the free tools available to you as a startup, as well as who to look for for that first researcher you hire, and how to scale the team as you grow.
So I hope that these tools are useful to you and enable you to better understand your users so that you can ultimately save time, reduce risk, make better decisions, and accelerate the growth of your business. And if you have any further questions, I look forward to chatting with you at the roundtables at one ten.
And like I said, please say hello to my colleagues at the Skyscanner booth as well. Thank you for your attention.