Measuring Security

There’s really no way to compare two computers running two different operating systems, Web browsers, or any other type of program and definitively say which one is more secure. That makes it hard for governments and businesses to decide how best to spend money on security–or even how much they should spend in the first place. It’s difficult to know whether a security product is effective or just has good marketing.

Consider virus scanners. They automatically examine files for malicious software, but they can only detect malware that’s already been identified. So a scanner can’t say that a computer system has zero viruses–it can just say that a system doesn’t have any of the viruses the scanner was designed to catch. But unknown pieces of malicious code have been responsible for many of the most devastating attacks to date, including the much-publicized attack on Google earlier this year.

It used to be that there were surefire ways to know your system had been hacked. Files would be deleted; attackers would alter your website or make your system crash and ask for ransom. Today, however, the goal is to steal information or take control of a computer without tipping off users. Because many attacks go unnoticed, there are no truly reliable statistics about how many computers are compromised, let alone statistics that can measure the full economic impact of these intrusions.

And yet people are trying. Research projects at the Idaho National Laboratory, the U.S. Department of Defense-sponsored Institute for Defense Analyses, and MIT Lincoln Laboratory are all attempts to develop ways of measuring security. If these projects can successfully create a set of standardized metrics, it will be easier for companies that create good products to reap a return on their investments in research and development, rather than competing on a level playing field with those who simply have a huge marketing budget and those who are selling snake oil. In the meantime, the attackers gain ground.