Fans of Apple computers often boast about superior security. But as Macs have gained in popularity over the past few years, this has brought much more attention from hackers. At a presentation scheduled to take place today at the Black Hat DC computer-security conference in Washington, DC, one security expert will reveal a technique for attacking the Mac operating system–OS X–without leaving a trace.

Similar techniques have targeted both Windows and Linux machines for several years. They allow an attacker to cover her tracks, eliminating vital evidence that an investigator might normally use to prove that a machine has been compromised; they might also allow the investigator to put together details of the incident. Bringing the technique to the Mac, however, required a significantly more sophisticated approach.

The technique that will be outlined at Black Hat DC allows an attacker to remove virtually all trace of an attack against OS X, after compromising the system using another exploit.

Vincenzo Iozzo, a student at the Politecnico di Milano, in Italy, explains that the technique allows an attacker to break into a machine without leaving a trace in its permanent memory, which means that evidence of the attack will disappear as soon as the victim’s computer is turned off. Such a technique could be used, for example, in combination with another software flaw to covertly replace a legitimate version of Apple’s Safari Web browser with a malicious one that logs the user’s keystrokes and sends them to the attacker.

Normally, when a user runs an application, the code runs in various parts of the computer’s memory. In OS X, a file format called Mach-O is used to specify where in the computer’s memory the application’s processes should run. Iozzo studied the Mach-O file format in order to predict in advance where these processes could be found. The technique identifies an active process (such as that for Safari) and injects malicious code into the space in memory where it is running. When the system reads from the expected location, it executes the attacker’s code instead of the legitimate program. Since the technique leaves no trace, Iozzo says that it can only be detected using software that watches for intrusions on a network.

Predicting where to inject the malicious code is made more difficult by a security feature in OS X that stores the variables needed to keep the attack untraceable in random locations within memory. However, Iozzo discovered a way to anticipate where the variables would be stored based on pieces of information that remain unchanged.

Dino Dai Zovi, an independent security researcher who specializes in Macs, says that Iozzo’s work is “very interesting,” particularly given the difficulties that he needed to overcome to make the stealthy technique work on OS X.

Dai Zovi says that, for now, there are few Mac attacks sophisticated enough to need protection of this kind. But he adds that the technique could prove an effective way to get past advanced antivirus software in the future.

Attackers haven’t focused much on the Mac to date because its smaller audience means smaller potential gains. But Dai Zovi notes that this is starting to change, and he says that researching the system’s vulnerabilities now should give defenders time to prepare for future malware.

Iozzo says that it may take time for Apple to respond to his technique because it exploits fundamental elements of the operating system’s structure that can’t be changed with a simple software patch. He says that it may require a larger upgrade, such as the introduction of the new version of OS X, called Snow Leopard, which is scheduled to ship in 2010.

In the meantime, Iozzo says that users can protect themselves by keeping their systems up to date with any security patches released for OS X. Since the technique relies on other flaws that an attacker might exploit, users should focus on reducing those other threats as much as possible, he says.

However, the technique could soon pose a threat to another kind of device. Iozzo says that he is currently working with another security researcher to extend his technique to the iPhone.