A client of our is thinking of purchasing comScore data. In the process they got a free trial of Nielsen//NetRatings which they asked me to look over.

I started by looking up the largest site we administer - netterimages.com, a site that sells rights-managed medical illustrations. Nielsen didn't have detailed demographic data on the site, but they did have estimated traffic at 76,000 unique visitors a month. Our stats software says we run closer to 125,000 unique visitors a month, so they under represented our traffic fairly substantially - even if you factor in different methodologies and the fact that Nielsen's data may have only been for the US.

But services like Nielsen and comScore aren't really intended to be used for sites the size of netterimages - they're intended for larger sites. So I figured I'd compare the netterimages data to the data on the two big players in stock and rights-managed images - Getty Images, and Corbis. Those are huge, well-established sites. Getty Images is a public company (GYI) with a market capitalization of $2.9 billion.

I was shocked when I saw the results...

According to Nielsen, gettyimages.com gets 299,000 unique visitors a month, and Corbis 86,000 unique visitors a month. That means netterimages.com has 88% of the unique visitors of Corbis and 25% of the unique visitors of Getty. That's just absurd. There is no way netterimages has 1/4 of the traffic of the site for a $3 billion company. It would be nice if it were true, but there's just no way it's true. Yes, we've optimized for Google Images and get a lot of traffic that way, where Getty and Corbis don't need to do that (and probably don't want to), but nearly the traffic of Corbis? If only...

I then went to Alexa and did a comparison and the results seemed much more believable...

According to Alexa, Getty has a 'reach' 38 times greater than netterimages.com (not 4 times), and Corbis has a reach 11 times greater (not 1.13 times). Those are much more realistic numbers. Yes, Alexa's 'reach' is not the same as Nielsen's unique visitors, but the two should be rough multiples of each other.

Thinking about it a bit further it struck me that if Nielsen is only reporting 60% of the traffic for netterimages and the gap between netterimages and gettyimages and corbis is greater than Nielsen reports, then the stats Nielsen is reporting for gettimages.com have to way off the mark - way, way off the mark.

Gettyimages.com has an Alexa rank of 1,261 (836 in the US) - it's a major site, and major sites are what Nielsen (and comScore) are supposed to 'get right' (since their sample size should be large enough to be accurate). But if Nielsen can't get their data even close to correct for a top 1,000 (US) site, then it raises serious questions about both the depth and accuracy of their data.

The bottom line is that Nielsen NetRatings doesn't even come close to being accurate, even on large sites like gettyimages.com. That raises questions about the quality of their samples (aka panels). If their samples aren't representative of the population at large, then their data has no statistical significance. Without statistical significance their data is worthless since no reliable conclusions can be drawn from a statistical analysis which is based on a flawed sample.

Update

As the months have passed and I've thought about it a bit I've come to realize why Netter appeared to do so well compared to Getty and Corbis... It's actually quite simple - Nielsen NetRatings is a survey of households, not businesses. While netterimages.com is primarily a B2B site, a lot of the traffic is kids doing homework assignments. Kids do their homework from home, hence it falls into the Nielsen's sample, but the corporate users who are the primary Getty and Corbis' traffic simply aren't in Nielsen's sample. Perhaps there's some home-based businesses who fall into the sample, but by and large those sites don't appeal to home users so they're under-represented in Nielsen's sample.

The bottom line is that Nielsen is probably a pretty decent sample if what you're needing is home usage. For example, this post was mentioned in relation to the strength of political web sites. Since voters would do much of their political web surfing from home, Nielsen would actually be an appropriate in that particular instance.

The problem is not that Nielsen is completely bad, but that they don't state clearly the limitations of their sample. As a result people make conclusions with Nielsen's data that they should never make...

Tags: alexa, nielsen-netratings, Web Analytics
Categories: Web Analytics