Select your localized edition:

Close ×

More Ways to Connect

Discover one of our 28 local entrepreneurial communities »

Be the first to know as we launch in new countries and markets around the globe.

Interested in bringing MIT Technology Review to your local market?

MIT Technology ReviewMIT Technology Review - logo


Unsupported browser: Your browser does not meet modern web standards. See how it scores »

{ action.text }

Adding it up: Although print, radio, and television still account for a large percentage of total advertising revenues, their share is decreasing as more and more ad dollars are spent online. But this isn’t necessarily a boon to new-media publishers who rely on display advertising, because “keyword” or search advertising dominates the online ad market; it now accounts for nearly half of all online ad revenues.

Panel Discussion
What’s wrong with existing methods of measuring Web audiences? Lots.

ComScore and Nielsen Online, a division of the Nielsen Company, are the established leaders in the field of audience measurement and the sale, to advertisers, agencies, and publishers, of the data that audience measurement produces.

These third-party audience measurement firms exist because the internal logs of publishers are notoriously unreliable in quantifying user activity on a given site. “When publishers use their log files, there are many limitations,” David Smith says. He says that the limitations of using these internal logs (a practice sometimes called “census measurement”) include, in ascending order of impact, overcounting individuals with multiple computers or Web browsers; counting “mechanical visits” by Web “bots” and “spiders” (for example, when Google crawls the Web to estimate the popularity of sites) as visits by real people; and overcounting individuals who periodically flush out the “cookies” of code that sites stash on browsers so that returning visitors can be recognized.

To create more-accurate audience numbers, ComScore and Nielsen Online rely on a methodology inherited from television audience research: the panel. Nielsen, for instance, has recruited nearly 30,000 panelists for its flagship product, called Netview. Panelists agree to have their Web browsing monitored through interviews and through “meters,” or spyware, installed on their personal computers.

But what worked with television doesn’t work nearly so well with the Web. “Panels are always problematical,” says Spanfeller, “but on the Web they’re super-problematical. Panels undercount by one-third to one-half.” In short, publishers simply can’t accept that their audiences are as small as panel-based measurements suggest they are.

Among the problems with panel-based audience research, according to both Spanfeller and Smith, is that it tends to undercount people who look at sites at work, because most companies’ information technology managers won’t install strange spyware on their computers. Sometimes, panelists lie to interviewers. Also, both say, there is a straightforward “sampling error” (what statisticians consider the misprisions that derive from sampling too small a portion of a general population): with as few as 30,000 panelists, the audiences of smaller sites are often grossly underestimated or missed entirely.

A final problem with panel-based measurements is that at the moment, neither Nielsen nor ComScore has itself been audited by an independent party. Who knows, both Spanfeller and Smith asked darkly, how valid the firms’ reporting methods really are?

Nielsen defends its panels. “I guarantee you, if our numbers were higher than the publishers’ server data, we wouldn’t be having this argument,” says Manish Bhatia, the president of global services at Nielsen Online. Bhatia notes that Nielsen does sell products, such as SiteCensus, that install software tags on publishers’ websites and measure server logs. “In combination with panels, they’re useful,” he says. “But panels are more reliable, they provide demographic information, and they tell you what people do after they’ve seen an online ad.”

For its part, ComScore also concedes that server logs have their place: they disclose which Web pages a publisher served, and when. But like Nielsen, the company insists that only panels provide an accurate measurement of audiences and their demographic makeup. “Servers don’t measure people,” says Andrew Lipsman, director of industry analysis at ComScore.

Why are Nielsen and ComScore so wedded to panels? According to David Smith, “The incumbents have a huge amount of money invested in their methodologies–and getting them to admit they have a problem isn’t easy.”

Roger McNamee is more blunt. “I understand why Nielsen is so bad,” he says. “But why isn’t there anything better? There’s a huge market opportunity for any venture capitalist who is willing to fund a system that audits actual traffic.”

“What we need is a third-party Omniture,” says Spanfeller, referring to the website analytics software that many publishers (including Technology Review) use to log their own traffic.

1 comment. Share your thoughts »

Credits: Toby Burditt, Chart sources: Jack Myers Media Business Report; eMarketer

Tagged: Business, Web, advertising, Web, online advertising

Reprints and Permissions | Send feedback to the editor

Jason Pontin Editor

View Profile »

From the Archives


Introducing MIT Technology Review Insider.

Already a Magazine subscriber?

You're automatically an Insider. It's easy to activate or upgrade your account.

Activate Your Account

Become an Insider

It's the new way to subscribe. Get even more of the tech news, research, and discoveries you crave.

Sign Up

Learn More

Find out why MIT Technology Review Insider is for you and explore your options.

Show Me