Numerous media outlets are reporting today that viewership of the American Idol premiere was down compared to past seasons. Reuters reported:
The two-hour Fox network broadcast averaged 4.4 million fewer viewers overall than last season’s premiere episode and fell by 13 percent in ratings for adults aged 18 to 49, the audience most prized by advertisers, Nielsen Media Research reported.
For the average man on the street, Nielsen Ratings are basically just a fun way to keep track of what’s hot and what’s not. For TV networks and advertisers, however, an increase or drop of just a few percentage points for a program can be worth tens of millions of dollars. Advertisers buy TV spots based on the size and demographic mix of a show’s viewing audience. If Nielsen says that your “prized audience” is declining, you won’t be able to charge as much per advertisement.
Prior to getting into advertising, I pretty much assumed that Nielsen Ratings must be based on some rigorous, time-tested methodology that derives a high degree of accuracy. After all, if billions of dollars of advertising change hands based on this data, it must be pretty powerful stuff.
As more and more ad dollars shift online, it’s only natural that Nielsen would want to provide similar traffic ratings for Internet sites. As a result, Nielsen now offers “Nielsen NetRatings” to it’s customers. A similar site, comScore Media Metrix also provides online visitor reporting.
And while I don’t have enough knowledge of the methodology to rate TV viewership to say with confidence how accurate it is, I’ve definitely had several anecdotal experiences with the online traffic measurement services to be very concerned with the results I’ve seen. Let me give you two examples.
A few years back I worked at FindLaw, an online legal Web site. Our marketing team worked hard to grow FindLaw’s traffic. After only a few years, our internal data was showing almost five million unique monthly visitors coming to our portal. Based on the data from comScore, Alexa, Nielsen NetRatings, and any other data we could find, we were far and away the most popular legal Web site – by a factor of about five to one!
Then one day I saw a press release from our rival, Lawyers.com touting Lawyers.com as the largest legal Web site. In fact, I just checked the site and they have an updated release with the headline: “For the third consecutive year, Martindale–Hubbell [owner of Lawyers.com] maintained its lead in 2006 as the #1 online lawyer directory according to comScore Media Metrix, Custom Reports*”.
When my boss Stacy saw this, she was aghast. We had never seen any evidence to suggest that Lawyers.com was anywhere close to our traffic. After several calls to comScore, a clear picture of the situation emerged. Take a look at that headline from the Lawyers.com site. In particular, look at the end of the headline: “according to comScore Media Metrix, Custom Reports*.” Custom Reports (asterisks), what does that mean, you ask? Well, if you look at the small print at the bottom of the release, it reads:
*comScore Media Metrix Data based on an annual 12-month average comparing 2005 and 2006 unduplicated visitor traffic to Martindale–Hubbell directories (Lawyers.com + Martindale.com) with FindLaw Directory.
That probably doesn’t mean much to you, but here’s what it meant to us: comScore took the entire traffic of two Web sites – Lawyers.com and Martindale.com – and compared it to the traffic from just a portion of FindLaw’s Web site – the lawyer directory, and from this data was able to make the statement that Lawyers.com was bigger than FindLaw. This would be analogous to taking the population of Canada and comparing it to the population of New Hampshire and concluding that Canada was bigger than the US.
So why would comScore do this? Well, remember, this was comScore “custom reports” data. Custom reports is another way of saying “a report our client paid us to run.” Lawyers.com pays comScore to run a report that just happens to paint a very rosy picture of their traffic vis-a-vis their competitor. comScore gets paid, Lawyers.com gets a great press release. Pretty ridiculous.
The second moment that made me question online reporting comes from Nielsen NetRatings. As the sub-prime mortgage meltdown continues, many pundits have wondered whether the decline in the mortgage market could have an adverse impact on the online advertising market. In particular, would display advertising and search publishers (Yahoo, Google, etc) see a big dip in revenue as advertisers decrease the amount they are spending for mortgage ads on these networks?
Well, according to Nielsen NetRatings, the amount being spent by these mortgage companies on online advertising is enormous. This post on a stock market Web site notes that Nextag and Countrywide (two major mortgage ad buyers) collectively spent $128 million on online advertising – in November 2007 alone!
The part of this report that really surprised me was that a Web site from my former employer – Adteractive – made it into the top ten. LowRateSource, according to Nielsen, spent $46 million in July 2007, and $24 million in November. If you average those numbers over a year, you end up with about $420 million on online advertising spend annually for LowRateSource, again according to Nielsen.
And while it’s true that I haven’t been working at Adteractive since 2006, I can say with a very high degree of confidence that Adteractive did not spend $420 million, $200 million, $100 million, or probably even $40 million dollars on LowRateSource advertising in 2006. So Nielsen is off here by at least a factor of ten or greater. Again, it’s like comparing New Hampshire to Canada.
So what went wrong? Well, my guess is that Nielsen has totally over-estimated the amount advertisers pay for their ads. A lot of the advertisers listed in their top ten spenders are buying run of network (RON) or run of site (ROS) remnant inventory. This kind of traffic can often get sold for as little as $.20 per thousand impressions ($.20 CPM). Nielsen, on the other hand, seems to be pricing this stuff at over $2 CPM – so 10X higher. And that doesn’t even consider whether Nielsen is also inflating impressions to begin with.
I can understand variances of five or six percent in statistical analysis. I suspect that Nielsen’s TV ratings must have that sort of margin of error. But five, eight, or ten times off of the real numbers? Me thinks we have a long way to go before we see accurate online metrics.
Addendum: As if on cue, Google and Yahoo are now fighting over the accuracy of Nielsen and comScore metrics. See Internet.com article here.
Blogation 