Moneyball: What’s in Your Sample?

• When talking with new research clients at NuVoodoo we’re often asked where we get our respondents, since we never ask stations for access to their databases.

Long before we founded NuVoodoo at the end of 2010, residential telephones had become accepted as a nearly perfect starting point for a research sample. You could reach over 98% of Americans that way. If you did an adequate job randomizing the numbers you were calling, you could rely on the samples you collected. Additionally, in our world, Arbitron contacted their sample via residential telephone. If you weren’t going to talk to us, you weren’t going to talk to them – so you didn’t count in our radio-centric world anyway. By 2010, however, residential telephones were on the way out, with fewer households connected that way and few of them ever answering a call from an unknown number.

Among the least welcome sounds in our lives today is the telltale “bwoop” when we answer a call from an unfamiliar number – that “bwoop” resulting from the computer-aided dialing system connecting the call to an agent in a phone center. Maybe you answered thinking it was a family member calling from a friend’s phone. Whatever the reason, this is assuredly someone calling about something that’s not important to you and reason to hang up the phone immediately.

The fact that many of the political polls we read about today are based on telephone samples should have us wondering, “Who are these people who answer their phones and answer the questions?” Should we trust estimates based on these samples? (We have come to the unscientific conclusion that they are either people with no lives or other researchers.)

Before starting operations at NuVoodoo, we spent years reading papers written by academics studying this very field. We did testing to see how the panels of online respondents available at the time performed in terms of gathering a representative sample. We learned how to manage respondents from the panels to build reliable samples for our work – and the results our client stations have gotten speak for themselves. As we started working on NuVoodoo, the internet had about 87% penetration. Today, it’s over 95%: comparable to the ubiquity of the old residential telephone number in terms of penetration.

In 2024, we employ databases from over 300 providers to gather samples for NuVoodoo research. While some of our competitors talk about using only “premium” sample providers or even a single, exclusive provider, we believe in having the respondents themselves make decisions for us. We rigorously check ALL respondents in our studies.

Tell us that you’re in one place when your IP address says you’re elsewhere? You’re not in our sample. Tell us you regularly listen to a station and then say a large percentage of the songs we test for that station are unfamiliar? You’re not in our sample. Unable to pass a few general logic questions laced through the interview? You’re not in our sample. If we get too many bad respondents from a specific sample provider, we stop using that provider.

Our JerkFinder™ software pores over results from every respondent to make sure they spent a realistic amount of time in the interview, that they weren’t making patterns with their responses, that they were sufficiently engaged in the interview. After that processing, a human being looks at each respondent and “I don’t like the looks of that one” is reason enough to discard. In the case of our library tests, the entire process is manual – we know how much is at stake for our client stations with these tests. We scrutinize the results before we release them to a client. Love everything (straight-line) or hate everything (no – you’re not a fan of that station) and you’re gone.

Following that process, a real, live human RADIO programming professional looks at every single report to make sure it makes sense, tracks for the format, trends with past results, and can be presented with confidence to our clients.

We’ve been told by our platform partner, who works with hundreds of research companies, that no one actually discards respondents and that our process is more strenuous than anyone’s in the business. We take that as a compliment!

Good research replicates. Asking the same people the same questions doesn’t count. Asking a similar group, drawn from a different sample pool, and getting similar answers is what we use to assess confidence in our results. This is why we access literally millions of different respondents to conduct our research.

While programmers today are asked to focus on a lot of things beyond just the music on their stations, getting the music right is table stakes to stay in the ratings. Music research allocations aren’t as routine as they once were. So, every test – be it regular testing of currents or a long-form library test – must be perfect. Even with inflating costs, we’ve kept our prices steady. We charge enough to keep NuVoodoo in business, while being mindful of what stations can afford. And we’re working with clients all the time to find solutions that keep the information flowing.

The old adage, “garbage in, garbage out,” is something we take seriously. For every project, every report, every client, every single day.

Need solutions? Get in touch with us at TellMeMore@nuvoodoo.com.