Moneyball: The Truth About “Rolled” Samples
• Talking to potential music research clients we regularly hear about their current research vendor giving them “rolled” samples. A “rolled” sample is one in which the latest sample, usually half the size of what a station would have preferred, is averaged with the previous sample. If a station gets new data every other week, the older half of the sample in the data a PD is reviewing is nearly a month old.
Rolled samples arose at the end of the days when radio stations were using telephone samples to conduct research about current music. As landline telephones became less universal, call centers became less and less efficient trying to gather respondents. Instead of each seat in a call center completing several interviews in one hour, it took several hours to complete ONE interview – pushing the cost beyond the reach of many stations. To bring costs into line with available budgets, some research vendors started the practice of rolled samples.
These researchers use “rolled” samples to reduce costs for themselves – to make it appear that they’re giving you the same number of respondents as they used to. They pay for the sample once, but use the respondents in two reports – in a sense charging twice for the same person. They do this while telling clients they’re reducing costs for them. They’ll tell stations that the numbers are more stable because of the larger sample. And the resulting research figures do in fact move very smoothly – but not because of the sample size. The rolled research numbers move smoothly because old numbers the station has already seen are averaged in with the fresh numbers.
The rolled sample scheme imagines that how people felt about songs nearly a month ago is relevant when making decisions about airplay of current music today. With current music, airplay across multiple stations in a market can result in burn scores spiking or passion dissipating. And with so many radio listeners now also exposed to new music on platforms like Spotify and TikTok, popularity can grow more quickly for some titles. To make the right decisions, you need the freshest data available.
Imagine if the speedometer in your car showed a “rolling” interpretation of your velocity. The readout would certainly be more stable. But how would that work when entering a busy highway? Are you at the speed limit yet? At a traffic light you might think you’re stopped, but the speedometer still says 30 MPH.
You need the freshest information available. Program directors today work hard – they shouldn’t have to guess whether a song is a hit today just because it was doing well weeks ago. How people feel about new songs is a moving target – they’re likely to feel differently today than they felt a month ago. Using old data in “rolled” samples risks making bad decisions.
NuVoodoo believes data-driven decisions are better decisions. We believe that if more stations were making data-driven decisions, more listeners would be better served by radio. We work to find ways to deliver great research at realistic prices so that more stations can have access to data for their decisions. And we deliver full samples for the same price as others sell rolled sample. We believe in unique samples for every report. It’s only right.
Rolled is great for lots of things. Joints? Flautas. California rolls. Lobster rolls are pretty great. A roll in the hay can be nice. For research programs involving current/recurrent music, “rolled” is not so great.
Questions? Hit up Leigh Jacobs at Leigh@nuvoodoo.com.
