Setting up research ops for customer discovery

There were a lot of sessions at our recent MTPCon APAC on Discovery. Really nothing represents the core of our craft more than connecting with customers and building empathy for their struggles. Following a round table, I thought I’d take some time to write down some recent tips/tricks that I’ve used for scaling discovery.

Lucky for us, research ain’t easy

In product land, we all hear the cliched advice ‘talk to your customers’ as if you can just talk to them and ask them ‘what do you want us to build?’ and they’ll tell us the right answers.

Fortunately what keeps us product people gainfully employed (but not so fortunate for innovation and progress in general) is that it’s not that simple.

Luckily there’s a method to our madness. We have at our disposal a great spectrum of user research methods. Christian Rohrer has a great map of research methods (below) so as a reference, we’re exploring the bottom left—discovery interviews.

A landscape of user research methods (Image: Christian Rohrer)

Discovery interviews

Discovery interviews can be extra helpful when your “next” customer segments are different to your “now” segments. Also adopting new problem space or partner platforms can be a good prompt to do fresh discovery research. I recently faced a situation where 80-90% of our research candidates needed to be found outside of our existing user base. If you’re working on a mature product then you’d be spending the majority of your time interviewing existing customers to improve retention.

It’s helpful to plan your research ops approach by considering the phases of the research pipeline:

1. Campaign planning (hypothesis, target profiles, recruitment mechanisms)

2. Recruitment (message crafting, searching, cold reach out)

3. Booking (calendar and appointment mechanisms)

4. The interview (principles around running them)

5. Analysis and follow up (note storage, candidate follow up)

I’ll explore these below.

Campaign planning

I’ll start a campaign by establishing as clear a profile as possible for who we want to talk to, what our assumptions and hypotheses are about them, and listing out example questions that may be used in interviews.

I’ve used confluence to set up a master page on the campaign because we’d end up attaching notes from each interview underneath the master page with various data points rolled up. I’ve since also found tools like Dovetail which are more purpose-built for storing research findings.

Our target profile would include mainly behaviours target candidates are exhibiting—for example the types of jobs they’re trying to do—followed by potential role titles they might hold.

We’d also brainstorm various ways we might be able to find these people including user database filters, app surveys, emails, LinkedIn searches, and other various lead sources.

Recruitment

I started out doing this job myself but realised it takes a lot of time and a skills set you’d find in business development or human resource recruitment. In pursuing this I used a LinkedIn Recruiter for message credits and for reaching out to people.

Good message crafting became an important element impacting conversion rate to interviews. For each campaign, I iterated at least a dozen different message versions for reaching out. What I learned (which ironically may be common sense to those working in sales/recruitment):

• Sometimes short is not always best – for those who reach the end of the message, their sunk cost in reading may influence them to engage
• Personal connections or references are number one – even if it’s the fact that they or you commented on a shared post etc, it helps
• Make the purpose extremely clear upfront – this is research, we’re not trying to sell you anything, you’re helping us
• We found removing the word “interview” and replacing it with “research chat” helped
• Make the call to action clear – e.g. a booking/calendar link
• Incentives do help – we called them “small tokens of appreciation for their time” – and offer where the target candidate is of a high perceived value
• Share your mission to change the world and how pivotal they are in helping advance that

Trying self-service research platforms

Recruitment itself was probably the biggest problem I had to figure out in our research ops. Being a product person, I also tried research platforms such as respondent.io, userresearch.com and askable.com. The way these platforms work is they have varying pool sizes of candidates (e.g. 100-500k) which they stock up via ads and other mechanisms from time to time. You can filter your research request demographically to certain profiles. 

The revenue model these platforms generally operate on is a referral fee which might be a percentage of the incentive offered. In using these platforms I found myself wanting more sophisticated filtering than demographic. They advised me to put a lot of work into my screening questions—such as asking people to answer freeform versus yes/no answers which they could “guess” to reach the incentive. 

I had to run a test campaign on each platform to gauge the quality of research leads. Although they may have the smallest pool size, I found Askable to have the best quality leads with the highest engagement. (We were generally looking for full-time office workers and ran this test campaign in Australia where both Askable and we are based.)

Asking friends and family, HR lends a hand, hiring an assistant…

The next step for us was to ask around the company for referrals. I liken this to an actual startup because sometimes the first customers you try to sell to are your friends and family. While the referrals are very strong, they can run out quickly.

For a while, we had someone on our people and culture team who normally looks after employee recruitment put in a day a week helping us with research recruiting. I think that’s a testament to great company culture when another department can pitch in like that.

We ended up actually creating a part-time—3 day a week role for research assistant—to mainly focus on this recruitment function for research for all the campaigns being run across the product team.

Booking

We used Calendly for our booking system. What worked well here was adding our photos to the booking page, an overview of what we were trying to do and being able to explain for privacy what we’d be doing with the data (and gathering acceptance). We used a team account and Calendly checked our individual calendars for availability. I also configured my event page to send an email and SMS reminder to the candidate the morning off and 1-2 hours ahead of our meeting time. This really helped our “show up” rate.

What could be better in this area is screening questions upfront. Due to our recruitment, 95% of the people we’ve spoken to have been a good fit. However, we could have filtered out a few non-fits with a screener questionnaire. We’re not sure about the impact on conversions this would pose.

Also what was slightly problematic was not being able to conditionally route bookings to a different calendar if needed. The fine product folk at Calendly know about this and it seems their automation logic may solve this in the future.

Calendly has a “round-robin” feature so now certain types of interviews are booked in “round-robin” style to our whole team. It checks the individuals’ calendar but interviews go into a central research calendar. This calendar is shared with the whole company. We announce certain types of interview books on Slack because we want to encourage anyone available to join the call. We’ll only take 1-2 extras, however.

The interview

Interview technique is a topic that’s been well covered in Mind the Product talks but we did establish some approaches and principles worth sharing.

Scribing

All of our bookings went into a central calendar so that one person from the product team (usually the person who launched the campaign) could be the interviewer / asking the questions and another person (anyone else from the company) could volunteer to “scribe.” We wanted to be respectful of people’s time and promised that we should be able to complete an interview in 20-30 minutes. The scribe would be taking notes live in a Google doc (live collaboration for the win) and the interviewer would be able to see the notes being taken. This approach was helped to quickly establish rapport and avoid multiple interviewers trying to dig into different areas in the interview. Sometimes the scribe would add extra questions to the document that the interviewer could ask. This helps the interviewer stay focused on the conversation.

Intro

To introduce the interview we’d remind the person what the company does, the purpose of our meeting being research and ask for permission to record the meeting (and begin recording if that was provided). Recording was very easy on Google meet—the recording would go directly into our Google drive where we were storing them anyway, we could easily paste the link into our company wiki. A really enjoyable workflow was the Google drive import -> Clipchamp edit feature where I could quickly clip a moment from the interview and then share on Slack. Google recently adjusted meet recordings to be on a higher pricing tier so we’re back to using Quicktime + loopback (on OSX) or alternatives like Zoom.

Not leading

We’d have a beginning set of questions that we’d work from loosely but more so be guided by trying to understand the significance of what the interviewee was sharing.

Try not to ask “do you want this feature?” Ask them about past behaviour because that’s a great predictor of the future. When done right, customer interviews can help you deeply understand their beliefs, expectations, problems and context.

For most interviews, after completing we’d do a quick mental retro or a quick retro between the scribe and interviewer to review our interview technique for improvements next time.

Analysis and follow up

After the interview, we’d paste the notes into Confluence. Confluence has a “page properties report” macro so we could put a small bit of data on each interview notes page and that could be “rolled up” and summarised at a campaign level. We capture consistent data across all our interviews such as the candidates’ persona, any competitors they mentioned, etc. We also capture answers to any questions that we gather for the campaign so the data can be rolled up.

The Dovetail solution we’ve also been trying has full transcriptions where you can tag certain quotes and build/share reports.

For good interviews, we’d post notable quotes, summaries and links to interview recordings in Slack for the team to consume.

We also wanted to be sure to follow up with the candidate. This meant sending out gift cards where applicable (managed in a spreadsheet), sending product coupon codes and requests for referrals to other interview candidates.

Where we’re at now is continuing to refine our research ops approach to ensure we have the systems to continue scaling and managing these processes. We’re currently considering a CRM type solution such as NetHunt, a flexible CRM integrated into Gmail. This could be helpful for us as the spreadsheets have gotten quite large.

You can see me talking through some of these tips during a ProductTank here.

Discover more content on User Researchor use our Content A-Z to find even more product management content.