We all know how important it is to keep the user at the centre of product development to achieve PMF and grow. But gathering and effectively leveraging user feedback to shape strategies and roadmaps can often pose a challenge – whether from a resource, skill or tooling perspective. As a result, we risk missed opportunities or unsuccessful product launches by failing to take the user perspective into account. Having run research at every stage of maturity from solopreneur to leading large teams at Facebook, booking.com and Skyscanner, Nyssa will share strategies for conducting user research in a relevant way for any size of business – from startup to big tech and everything in between.
Slides
Transcript
Thank you for joining me. My name is Nyssa Packard and I lead the insights team at Skyscanner, which consists of data science, consumer research and user research.
Given my role, I naturally love asking questions, collecting data, and sharing insights with people to help them make better decisions to fuel business growth. So there’s going to be a lot of information we’re going to work through.
This is something really near and dear to my heart because as a former founder, I’m very passionate about helping small businesses succeed. I hope that you’ll come away from this talk with lots of practical tips that you can apply to help fuel your business, no matter where you are in the growth curve.
What is user research?
Who knows what user research is?
It’s about understanding user needs, motivations, and behaviours. We do this with a variety of different methodologies which can be tasks, observations, or other types of feedback. So that’s all well and good. But why should you care?
Ultimately, it’s not just about understanding user needs. It’s about making sure that your business stays in business.
According to an analysis of over 300 failed startups, it turns out that half of those top 20 reasons why startups fail have to do with the customer. The number one reason why startups fail is that they didn’t understand the customer need that they were solving and there wasn’t a customer need for their product.
So it’s critically important that we take the time to make sure we understand what that user need is and what that customer need is to make sure that we will achieve product market fit.
The Harvard Business Review did an article on why startups fail and found the same thing and went further to say that by neglecting to research customer needs before commencing engineering efforts, entrepreneurs end up wasting valuable time and capital on MVPs that are likely to miss their mark.
Entrepreneurs are like sprinters who jumped the gun. They’re so anxious to get their product out there, that they fail to make sure that they’re meeting a need in the market.
Does this resonate with anyone anxious to get their product out there? I know when I was a founder, I was very keen to do so myself. So it’s something that we were all susceptible to, and something I still see at some of the biggest tech companies.
Ultimately, user research isn’t just about understanding needs and motivations. It’s about saving time and helping you accelerate so you don’t have to test and learn for nine months before you get there.
It helps you save money on engineering time, and design time, that doesn’t need to be spent until you first know what you’re solving for. And it ultimately reduces the risk in the decisions that you’re making. User research helps you reduce the risk in the decisions you’re making, so you can make better decisions and ultimately grow your business.
It’s easy to get tricked into thinking that your thing is cool. I think we all can recognise that you have to pay attention to your customers and adapt to their needs.
We often can get into the mindset that we are our end users and end up building for ourselves. That is not the case. That’s why you need to go out and talk to actual users or customers.
How does user research work at Skyscanner?
At Skyscanner, the mission of the user research team is around delivering actionable insights that improve user experience and accelerate business growth. And we have two key jobs to be done.
The first is about landing the current product roadmap as successfully as possible with our users. That’s typically what we call evaluative research, which I’ll get to in just a minute.
The second is about identifying the greatest opportunities for future business growth. What is that untapped opportunity to meet a need better than others better than competitors do or in a way that we don’t today? So identifying what should be on our future roadmaps and that often comes from what we call generative research.
I’ve been there with limited budgets. You need some cheap options available, as well as what to look for in the first user researcher you hire and how you can build out that team successfully as you scale.
So you probably have a product development flow similar to what we have at Skyscanner that starts with the discovery and inception phase, goes through conception and ideation to building a plan, a minimum testable product, or maybe an MVP to test or experiment, launch and monitor it in the long term. So where does research fit into this equation?
In the early days of discovery and inception, we do generative research, which might address questions like how can we achieve a product market fit. How can we simplify our information architecture? How can we improve customer retention? Or where are the future growth opportunities for us and new verticals or new markets?
Once we have something to evaluate whether that’s an idea, a concept, a prototype, or even an existing product, we can move into evaluative research.
There we can address business questions like how can we reduce cart abandonment? How can we improve conversion? Which new features should we prioritise? Or how can we improve the launch of our new feature to land as successfully as possible?
Then through experimentation, this is not often led by user research. But sometimes user research plays a role. In assessing how much the change we made moved the needle? Or in the monitoring phase?
What should we focus on to optimise the user experience moving forward? This can be through tracking research or looking at analytics data.
Then we start the cycle again. It’s a circle. And not all decisions are created equal.
Conducting research is about reducing risk and saving time and money. So ultimately, the more risk that is associated with a decision if you’re changing your checkout flow, and payment options, redesigning your homepage or doing something that requires a lot of engineering effort like building a new app. These are the types of things that are most critically in need of user research.
On the other hand, we might be optimising tactically. Like testing different colours of buttons, different text options, and adding different filters. Those are not necessarily things that need research, you can just A/B test. But it’s the big strategic bets that you want to make sure are informed by your users.
How do you get started on user research?
How do you get started if you haven’t done user research before, especially if you’re budget constrained?
You always need to start with your business objective. That sounds very obvious. But I can’t tell you the number of times that research projects go wrong, because this wasn’t clear from the start. So get crystal clear on what you’re trying to achieve.
For example, we might want to say how we can improve the number of customers that sign up so that our sales teams don’t have to manually go out and sign up customers. So we want to increase signups of new customers via a signup flow.
The second is to identify the research questions. So that’ll be based on your business objective. In this example, that might be where do people get stuck in the signup flow, and why? From that clear research question, we can then identify which type of data we need to collect.
With this example, we want to understand the signup flow, looking at the analytics data, where do we see the biggest drop-offs? And then secondly, we would want to do user research, contextual inquiry, or usability testing, to have people go through that signup flow and see why they’re getting stuck at those places.
What did they expect to discover? What information do they not have available? Or what did they misunderstand?
There are lots of different methods that we can pull from in our toolbox as researchers. And we typically divide these into a couple of different categories.
- Qualitative and quantitative would be methodologies where we’re talking to typically less than 10 users per subgroup that were interested. So that would be things like interviews and focus groups. That’s the type of attitudinal research where we’re asking people about what they want, what they need, and how they feel about a current experience.
- Then we have behavioural methods where we’re looking at how people behave. That would be things like ethnography, where we observe people in their natural environment and see what they do. Or usability testing, where we give them a task, they’re engaging with your product, and you observe how they interface with your product.
- We also have tools like diary studies to look at that longitudinally over a series of weeks. For example with travel planning, that’s often that often spans several weeks. We use eye tracking to see where people are looking on a page, what they’re missing, and what’s drawing their attention. There are also tools like card sorting, where we can look at how things should be grouped to inform your information architecture, or to create bundles for products.
- Then when we head into the quantitative side of the graph. We’ve got hundreds or even thousands of responses from users. That includes things like data mining, and AV testing, which is in the millions or more data points. We’ve got surveys down in the attitudinal side of things where we’re asking large numbers of people what they would like, what they prefer, what they need, or how good a service was.
- Co-joint is a more sophisticated technique where a series of options are given and based on the choices they make, it extracts the importance of different elements from those choices, different features, different price points, etc.
This is not comprehensive, but that gives you a bit of a foundation of the types of research methods available.
How can you get started for free if you want to start doing research as a startup?
These are actually the tips and tools that I applied myself. For things like interviews and focus groups, you can recruit people with Calendly, which is a signup tool. You can set the time slots, people can sign up, and go to Zoom where you can record the session. And boom, you have your session recordings of interviews to analyse.
If you want to not do remote testing, but you want to intercept people and have face-to-face engagements, you can do what we call guerilla testing that has nothing to do with primates!
It’s where you intercept people in a likely environment for them to be in. So if you have a fitness app, you might want to intercept people outside of a gym. Then you can ask them for feedback on whatever topic you need information on.
We also can use tools like Miro for card sorting. It’s a virtual collaboration platform with virtual posts that enable people to group different ideas and features into the most logical mental groupings.
In terms of behavioural options, of course, we have dogfooding which a lot of us are familiar with where you put yourself in the shoes of your user and go through the user journey yourself.
At Skyscanner, that would mean we would book a hotel or flight for ourselves. As we go through the process we look for what’s confusing, where there’s friction, and where something happened that was unexpected. You then make note of those things to help improve your product.
If you’re at Uber, this could be a drive-along where you sit in the passenger seat, and watch for the day. What’s the experience like for the driver or the passenger, etc?
Then there’s a great tool that I love called Hotjar that spans the behavioural spectrum from qualitative to quantitative. They do session recordings of what the user does on your site. It doesn’t record any personally identifiable information but it’s also unbiased behaviour as people don’t know they’re being recorded.
So you can see where they actually get stuck, where they’re rage-clicking, where they can’t fill out a form, or whatever it might be. They also aggregate this data at a quantitative level so that you can see heat maps of where people are clicking, and what they’re missing. It kind of simulates eye-tracking data and how far people scroll.
In the analytics quadrant, we have things like Google Analytics, Mailchimp, Google ads, and Facebook ads; a lot of things I’m sure you’re using to get your AV testing data on product changes and marketing campaigns.
There are a couple of tools that you might be less familiar with, like Google Trends, and Facebook Audience Insights, which zoom out a level further in terms of data. It’s not data about your product or your marketing, it’s data about the market at large.
Google Trends looks at all of the search data on Google so you can see what types of topics are trending, what the search queries are around that topic, and where geographically those searches are coming from. It’s a wealth of useful data.
Facebook Audience Insights uses all the profile data across Facebook and Instagram to help you get a better understanding of the profile of people who like a given topic or use a certain brand.
When we get to surveys, you have a couple of options. There are many free survey tools like Google Forms. The challenge is how do you get those surveys distributed to people to answer your surveys.
You can send it out through your social media channels to your followers, that’s totally free. Or if you want to reach a broader base of people who are of a specific demographic, do that as a boosted post, or even run it as an ad. This enables you to do the same type of targeting you would for your ad campaigns, get women 18 to 34, who live in a certain region, whatever you like.
This is the suite of tools that you can use to get started. I do recommend looking at some how-to guides because it’s not just a free for all that you use the tools and you’ll be good to go. You need some kind of understanding of how to do it right.
How to use these tools in practice – an example
To give you an example of how I put this into practice. My business objective was ‘how could I build for product market fit’. I had done some studies, I’d worked in ecommerce and I wanted to start a wine ecommerce business, but I didn’t know what type. So I wanted to get to that winning idea. I started with market analysis.
Using Google Trends I looked for which types of countries had the most growing demand in terms of interest in wine. What types of wine were they interested in? It turns out Prosecco in the UK continues to be increasing in popularity every year. And non-alcoholic wine was a big trend. If anyone’s doing non-alcoholic wine, there’s a big opportunity out there for you.
So research the wine market in general. Then I looked at competitors, using Facebook Audience Insights to see what is the profile of people who are currently buying from brands like Naked Wines. What are they interested in? It turns out nesting and travelling. So that helps me with my marketing campaign to know what interest areas I can piggyback on or what kind of co-branding I can do.
I then went through the critical stage of looking at the jobs to be done and using the value proposition canvas. So how does that work? For those of you who don’t know, it’s a methodology based on what people are trying to accomplish and the theory that they hired different products to meet those needs.
The job in my case was not about buying wine, it was about enhancing a meal, impressing guests unwinding, and even being perceived as refined.
That’s what people are trying to do and they can do that with wine, cocktails, or a book. There are many different products that can meet the needs of that job.
From that job, you look at the pains and gains. What are the things that help people achieve this job? What are the things that can get in the way and make it difficult?
For me, one of the things I heard was not knowing about wine and the risk of getting it wrong and ruining a meal or embarrassing yourself in front of friends.
From there, you can go into shaping your value proposition, which means looking at which gains can you enhance, or which pains can you reduce. For example, having a curated selection of wines done by an expert, pairing tips for which food to pair with your wine, and even some recipes that will make it go perfectly.
Ultimately, that’s how I got to my value proposition around a small business-focused wine subscription. From there, if you unearth several different jobs to be done, which is likely you can size those by looking at how big are those jobs in the market, how many people are trying to achieve that job, and how satisfied are they with the current options available.
We’re looking for the sweet spot of jobs that are sizable and underserved in the market. Because that means that’s low-hanging fruit for us to go meet the needs of these people.
From there we can start to ideate. So this is the concepting phase, where we can test out different ideas and see how appealing they are to our target demographic, as well as how unique they’re perceived to be relative to the other competitors.
These are some of the concepts that I actually tested for my business. It’s a snapshot from the focus groups I ran at my house, I also did interviews. Like I said, all free. And tried out natural positioning around organic and sustainable, a world tour about the discovery of different wine regions in different countries.
The winning one was the hidden gems around niche boutique wines that were hard to find from small family vineyards. So that was what I moved forward with. Then we’ve got the trifecta as the end product.
How do you then build a product?
So we’ve now assessed the desirability from the research, but we need to look at the feasibility. This is where you need to talk to your engineering partners and say, can we actually build this? How can we build this?
And then there’s the viability about profitability. Can we make money with this? How much are people willing to pay and how much will it cost us?
How much people were willing to pay was another thing that I gathered through research. How much will people pay for a three-bottle box and six-bottle box or twelve-bottle box, etc?
I also tested things like the branding to look at what was appealing to people and what was memorable. Which brands could they recall after a five-minute survey spontaneously? This meant it would be easier for them to communicate it via word of mouth because they could recall that name.
You can see this is how I ended up with my brand, the most memorable vino, as well as one of the most liked the Curious Wine Club, which gave me VYNO. It’s a bit nostalgic looking back on this and I can commiserate with the amount of stress and uncertainty that a lot of you are dealing with right now.
Unfortunately, I had to close down my business because I was diagnosed with cancer. I’ve been in remission now for four years. But the stress that you go through is very palpable. I’m hoping that these tools will help you to make better-informed decisions to help fuel your business growth.
How to hire your first researcher
Now your business is continuing to grow, you’re ready to hire your first researcher. Who should you hire and how do you get started?
To set yourself up to succeed, you need to hire an experienced user researcher. There might be times when you hire someone straight out of university, but this is not the time. You need someone who knows what they’re doing, can hit the ground running, and be autonomous.
You also don’t need the most experienced person who’s led teams because you want someone who’s going to get their hands dirty and be a bit scrappy.
You also want to set priorities for the person coming on board and the rest of the team about using research moving forward to help make your decisions throughout product development. And setting the priorities for the researchers again about those business objectives.
What are the key things that they should focus on solving? Because inevitably, what I hear from many researchers who are the first is they get inundated with requests, and they don’t know where to start. So it becomes about who’s shouting loudest? Or who do they have the best relationship with? And that’s not how you want them to be prioritising their time.
Perhaps the most important piece is about integrating this into your product process. So you saw what our flow looks like at Skyscanner. Think about how you communicate this to your team and your flow, about when and where you’ll be using research to inform your decisions.
Who should you be hiring? This is a great analysis based on 70 different user research job postings from the past year. You can see the top piece that’s important is the methods that this person has experience with. And in terms of methods, the most important methods they really need to know are surveys, usability testing, field research, and interviews under their belt. Then they’ll be able to cover the majority of the needs your business might have.
Beyond methods, we have things we would expect from most team members, things like communication, project management, and collaboration. Then we get to things like experience and qualifications.
Having had people in my teams with backgrounds like history, foreign languages, and English literature, it doesn’t really matter what they studied. I think having years of experience is more important.
So you’ll probably want to target that three to five years range, which is a sweet spot in the market.
How do you build the team to scale with your business growth?
You’ve got that person started, there’s a lot more momentum. People want to learn more from your users. So how do you build the team to scale with your business growth? You’re likely to start off with one of two models.
This one I recommend is the agency model where the team will work on projects like an agency for different product teams across the business. That means that they need to prioritise ruthlessly on the most important things, the most important questions for the business to answer.
This is how we approached it at Skyscanner recently because as you can imagine, coming out of COVID, the team was pretty depleted. So we had only one or two researchers at one point. And we needed to move to operating like an agency model for a while.
So we took all the research requests we would receive, bundled them into requests that could be handled in one project, and prioritised those with the C-suite to make sure we were working on the most important requests for the business.
It seems very simple, but If we hadn’t done this and taken that first in first out approach or prioritising based on who’s shouting loudest, we probably would have only taken on three projects instead of the twelve we were able to take on by bundling and prioritising effectively.
The other option you have is a democratised model. This would be having that same small set of user researchers at the core but using them more as mentors to help PMS and designers across the business conduct their own research.
So there are some advantages to this in terms of you can scale the amount of research you can do very quickly because you’re just training up your existing workforce to be able to get started. But there’s intrinsic bias in the research that will be done by PMS and designers.
Not because of any ill will, but because it’s their idea, their design, and their baby and no one wants to hear their baby is ugly.
So you’re intrinsically not going to be very objective when you’re testing something you have worked on and put blood sweat and tears into.
Secondly, it takes time away from the work they would otherwise be doing from building product, designing, etc. So this is an option I’ve seen businesses apply, but it’s not my recommended approach.
How do you scale?
As you continue to scale, there are two options there as well.
You can try an embedded approach, which I’ve seen in some businesses, where you have user researchers reporting directly to the head of product for their area. This gives them a contextual understanding of the product space and enables them to be proactive in identifying what other research might be needed.
The downside is that they operate in silos, so you might get duplication of work across the squads. And you lack that connective tissue to inform strategic insights that don’t fit within one part of the product but are the bigger-picture business-type insights that you need.
It’s also not great from a personal development perspective, in that people aren’t getting that mentorship and exposure to how others work to grow in their careers.
My preferred model and what we’ve applied at Skyscanner is a hybrid approach, where we have user researchers dedicated to each of the product teams. So they have that context and they’re able to be proactive. But we also have a core, centralised reporting line, which avoids duplication.
It enables personal development and taking a holistic strategic view across all of the work that we’re doing. We also have champions that we empower across the different product teams to do their own self-serve research, with templates, and how-to guides. And a shout out for research ops as well, which is a key enabler of efficiency making the best use of your team’s time.
So we’ve gone through why research matters, how you can get started, what kind of questions can be addressed with research in your business, the free tools available to you as a startup, as well as who to look for, for that first researcher you hire, and how to scale the team as you grow.
I hope that these tools are useful to you and enable you to better understand your users so that you can ultimately save time, reduce risk, make better decisions and accelerate the growth of your business.
