Scientists have pretty good methods for surveying, say, voter intentions. In that case, you focus on getting a good representative sample. Inaccuracies matter, but a few one way or another aren’t going to make much of a difference.
Cybercrime is a whole different story. For one thing, cybercrime surveys are trying to measure a number: how much money was lost. In that case, individual responses can make a huge difference. A voting survey isn’t thrown off by much if someone who actually plans to vote Democrat states an intention of voting Republican. But if a survey respondent who has lost $50,000 to cybercrime claims to have lost $500,000, any calculations based on that information will be wildly out of whack.
There are other problems. Any registered voter has useful information to report. But not everyone has a useful story to tell about cybercrime, which means that a small number of responses can make a huge difference. For example, in a 2006 survey conducted by Gartner Research, 128 out of 4,000 people claimed to have been victims. Herley calculates that 59 percent of losses came from the top 1 percent of respondents who had been victimized—in this case, a single person. He believes that such concerns make it impossible to trust data coming from most cybercrime surveys.
“Understanding the impact of any crime is problematic,” says Julie Ryan, lead professor in information security management at George Washington University. However, Ryan says, cybercrime presents particular problems because most people aren’t well equipped to answer technical questions about it. For example, individual survey respondents might not be sure whether they’ve been victims of a phishing attack or, indeed, whether anything was taken from them as a result. “So here we have a problem,” Ryan says. “Potential crime that is potentially undetectable, compounded by a target space that is mostly ignorant.”
Compounding the problem, it’s hard to get unbiased information, Ryan says. Big corporations might be reluctant to admit to being cybercrime victims, since such an admission could lose them customers. On the other hand, security firms—which often conduct surveys on cybercrime—benefit when people take cybercrime seriously.
Herley wants companies and organizations that conduct surveys to publish much more information, making it easier for researchers to evaluate their methods. For example, they could publish median figures for cybercrime, which aren’t as likely as mean, or average, figures to be distorted if a few people exaggerate or are mistaken.
Herley also suggests that claims of big losses receive careful scrutiny. “You risk catastrophic error,” he says.