I used to manage research programs for Google, GEICO, and Facebook. I spent millions of dollars on research projects and hired over a dozen different research agencies during this time. Some of these agencies were large, global companies. Others were small, boutique firms.
It might sound like fun to write checks all day long (especially when you have a Google-sized budget). It might also sound fun to hire some of the “best” research agencies to do interesting work on your behalf. But, in reality, the life of a corporate researcher can be quite stressful. When you spend a large portion of your time managing vendors, your success is dependent on the quality of their work.
If your vendor produces research that is insightful and it guides long-term strategy for the business, you might find yourself on an internal “road show” as you tour the company to share the research results with executives and decision-makers in a wide range of departments. A year later, this specific project could be listed as one of the key factors behind your promotion and/or bonus.
If your vendor produces research that is a dud, then the report will generally sit on a shelf somewhere, collecting dust. This might not lead to you getting fired, but it could lead to people within the company feeling like research is just not worth the investment. People might complain that research only tells them what they already know: that water is wet.
As a corporate researcher (aka a “client”), there are things you can do to improve the quality of the work from your research vendors. In other words, you can do more than just write checks. First, you can put your vendors through a careful vetting process to make sure you weed out anyone who doesn’t actually understand research. Second, you can hand your vendors a tight brief so they have a clear understanding of the research objectives and applications. Third, you can work alongside your vendor, reviewing their materials (survey drafts, focus group stimuli, etc.) to provide feedback regularly during the project. Fourth, you can review the final presentation before it is delivered to stakeholders. Many corporate researchers will actually receive a presentation from their vendor with final results during a 1x1 session. Afterwards, the corporate researcher will totally re-do the presentation (to simplify the story and perhaps make it look more “pretty”) before sharing the results internally.
But, as a corporate researcher (aka “client”) there are certain things that are beyond your control. There are things your research vendor might do that ultimately “doom” your project. The vendor doesn’t want to see you fail and they don’t want to produce lackluster research. But, they are often in over their heads in terms of research and too focused on profit margins. So let’s talk about some of my pet peeves when I was a client managing research vendors. These are things that research agencies do that directly drive down the quality of a project and, in my opinion, aren’t entirely honest.
The first, and most annoying issue, was the “bait and switch.” Here’s how it works. During the pitch process, a research agency brings their best and brightest researchers to the room. You find yourself surrounded by PhDs and veteran researchers with decades of amazing experience. The team is impressive and they seem to understand the challenge you’re facing. So, you sign the deal, and kickoff the project. Suddenly, the “superstar” researchers at the agency have disappeared.
You thought you were hiring a team of people with 20+ years of experience. Instead, you find that the people working on your project are much more junior. Perhaps a person with 10 years of research experience is “managing” the project but the real people doing the work are analysts with only 2-4 years of research experience. You’re assured that the more senior people are involved “behind the scenes” but the quality of the work (survey drafts, analysis, presentations, etc.) seems to suggest otherwise. There are obvious errors in methodology and the research lacks the intellectual curiosity and analytical rigor that is needed to produce insightful research.
I have experienced the “bait and switch” on $45,000 projects and $500,000 projects. It seems to be a common practice. It is not only annoying, but it leads to poor quality research. I don’t have anything against junior analysts – I was one once. But without the right supervision, training, and mentorship from a more senior researcher, junior analysts are often left flailing in the deep end of a project. They can’t produce high-quality research because they don’t know how to do so.
To get around the bait and switch, I would encourage you to ask your vendor who will be working on the project and what percentage of their time will be dedicated to the work. Your vendor will not want to answer this question directly, but will instead talk about a “team effort” that involves lots of individuals in the company with no specific commitments from any one person. Push your vendor to explain whether or not key aspects of the project – like survey design – will be handled by a more experienced researcher or a more junior analyst. Who will write the first draft? Who will review?
If you hire a vendor and you’re worried that they may be using the bait and switch technique to dump your project on more junior analysts, ask very specific questions in your weekly check-in meetings. Pay careful attention to who is able to answer these questions. If the Director of the project (who is supposedly managing and reviewing everything) has to regularly turn to the analyst to answer your specific questions, you’ve got a problem.
Some other issues that annoyed me when I was a corporate researcher (aka “client”) managing research vendors were as follows:
Sample Sizes – Vendors suggesting very low sample sizes for consumer surveys and calling the data still “statistically relevant.” Vendors will often decrease project sample sizes to unacceptably low levels just so they can hit a certain price point with a certain level of margin attached. You will need to be the “sample size police” when reviewing proposals from vendors.
Templates – Some research projects, like brand trackers and ad testing, can be relatively generic. You don’t need to re-invent the wheel every time you conduct a study. You can simply use templates from prior projects. But many of my research projects over the years have been custom. They are strategic research initiatives that explore uncharted waters. In these scenarios, you can’t rely on a template. You have to start from scratch. There are some vendors that possess the intellectual curiosity to create a truly custom project and there are others that do not.
Decks – Why do so many vendors create final PowerPoint decks with 100-200 slides? I have never met a corporate researcher who wanted that much information. Most of the executives I work with don’t want more than 10-20 slides. They expect their research vendor (or internal researcher) to digest all the information and focus on only the most relevant findings. There is a big difference between a story (with a recommendation) and a data dump.
Cross-sell – Sometimes you invite your vendor to executive level presentations. You want to include them so they can hear feedback from management. And you want to have them on-hand in case there are very specific questions about the data or the methodology. The worst thing you can see in one of these meetings is your vendor handing out business cards with fervor. I understand the need to win new business, and build connections within a company, but it can feel a little weird when it looks like your vendor is more concerned with selling their next project than wrapping up the current project.
If you ever want advice on managing research vendors, we're always happy to talk shop. Despite the frustrations I have experienced over the years, I have tremendous respect for our peers in the industry. Feel free to shoot us a line at firstname.lastname@example.org
Copyright © 2019 Honest Data - All Rights Reserved.