Over the past decade, Microsoft, the target of choice for many online attackers, has hardened its operating system, adopting technologies designed to make it harder for attackers to find and exploit vulnerabilities. Apple and many other software makers have followed suit, introducing similar additional security measures to their operating systems.

Yet last week, during the “Pwn2Own contest” at CanSecWest, a security conference in Vancouver, Canada, security researchers demonstrated that software makers need to do more to protect their programs. Using previously unknown vulnerabilities, the researchers were able to compromise Apple’s Safari, Microsoft’s Internet Explorer 8, and Mozilla’s Firefox Web browsers by circumventing the latest security technologies in place in the operating system underneath.

“These things make it hard–they really do,” says Charles Miller, a principal analyst at Independent Security Evaluators and the researcher who circumvented the security of Apple’s Safari browser and the Mac OS X Snow Leopard operating system underneath. “But, no matter what, a determined attacker can find a way in.”

The results of the Pwn2Own contest underscore a truism in security: Defenders must be right all the time, but attackers only have to be right once. “The exploits are really creative; that’s why they are tricky,” Aaron Portnoy, security research team lead for TippingPoint, the security firm that sponsors the Pwn2Own competition.

Starting with its Trustworthy Computing Initiative in 2002, Microsoft began implementing a series of security technologies in Windows. First, the company protected “the stack”–the logical memory space used by programs to temporarily hold data. A technology called the “/GS flag” (after the software switch used in the company’s compiler), prevented attackers from pushing their own code to the stack. But in 2003, David Litchfield, an independent researcher, demonstrated a way around the protection. Microsoft reacted by rolling out two more technologies: SafeSEH to address the attack using structured exception handlers (SEH), and address space layout randomization (ASLR), to make similar vulnerabilities more difficult to exploit in the future. Researchers have, however, found ways around both those protections.

Most recently, Microsoft brought out another technology, data execution protection (DEP), which prevents attacks that overwrite memory with code and then try to execute that code. But earlier this year, an independent researcher, Dion Blazakis, showed off an attack, known as JIT spraying, that uses vulnerabilities in other programs–most notably Adobe Flash and Sun’s Java–to bypass those protections.

“These exploitation techniques are a hot commodity right now,” says Portnoy. “If you have a way to bypass the (operating system’s) security, then you are a step above most of the people here.”

Apple has not been immune, either. The company has continued to release more security technologies in its own operating system, and Snow Leopard includes both ASLR and DEP, according to Miller.

Microsoft acknowledges that software bugs will always exist, and says that the goal is to make exploiting such vulnerabilities less damaging. Today, other measures, including further stack protections, ASLR, and DEP make it harder to find and exploit vulnerabilities.

“If those techniques weren’t around, you would see a lot more exploits than what we are seeing now,” says HD Moore, chief security officer of Rapid7 and the director of the Metasploit Project, which packages exploitation techniques in an easy-to-use framework for security researchers.

Research on additional protections is ongoing, and a leading candidate is “sandboxing”–a technique where untrusted code is run in protected areas of memory and processing space and not allowed to affect other parts of the computer or device. The Java programming language and runtime environment made sandboxes popular, but only recently have programs been using sandboxes more extensively. Browser makers are looking at running their code in a sandbox, and Google’s Chrome, which survived the Pwn2Own contest without being hacked, runs code in a sandbox.

Moore says sandboxes do have their limitations. “Sandboxes are really good at protecting against a vulnerability in an application becoming an exploit of the operating system,” he says, “but it is only useful if the data that you are trying to protect is not accessible.” In many cases, the program may need access to sensitive or system data, and then sandboxing no longer helps, he says.

In the end, software makers have made their programs harder to exploit, says Miller. While he found nearly 20 vulnerabilities in popular software, such as programs created by Adobe, Apple, and Microsoft, less than a handful could be exploited on an up-to-date system, he says. “It’s a trade-off,” Miller admits. “Every time you add one of these (protections), it slows down the system or makes development harder. The goal is to make software hard to exploit, and they have done that.”