Networks of compromised computers controlled by a central server, better known as botnets, are a Swiss Army knife of tools for online criminals. Hackers can use these co-opted systems to churn out spam, host malicious code, hide their tracks on the Internet, or flood a corporate network to cut off its access to the Web.

Whenever a new botnet appears, researchers race to reverse engineer the software it installs on a victim's machine, and to decode the way each bot communicates with the controlling server. Because these communications are often encrypted, such analyses can take weeks or months. Now researchers from the University of California at Berkeley and Carnegie Mellon University have created a way to automatically reverse engineer the communications between compromised computers and their controlling servers.

In a paper to be presented this week at the Association for Computing Machinery's Conference on Computer and Communications Security, the researchers show how automatic reverse engineering can decipher the structure and purpose of the communications between a command-and-control server and its bots.

"The communications protocol of the botnet is the core of the botnet," says Juan Caballero, a PhD student affiliated with both the University of California at Berkeley and Carnegie Mellon University, and lead author of the paper. "That is how the attacker sends commands to the botnet."

When researchers have previously tried to automatically analyze botnet communicationprotocols, they focused on deciphering the commands received by the client. Yet Caballero, together with UC Berkeley assistant professor Dawn Song and two other colleagues, has developed a technique that translates both the commands received by a client and the responses it sends.

The researchers then ran the botnet code on a virtual machine and analyzed the movement of information to and from a computer's registers--memory components within a machine's processor--before it was encrypted. Watching for changes in the memory registers--the researchers call this "buffer deconstruction"-- allowed them to derive the structure of the botnet communications and infer the function of the various components of each command.

"This is relevant for malware, because we typically do not have the executable for the command-and-control server of a botnet," said Paolo Milani, a postdoctoral researcher at the Secure System Lab at the Vienna Institute of Technology and author of an earlier paper on automated protocol analysis. "So with previous techniques, we would not be able to automatically reverse engineer the client side of the protocol."

The researchers built the resulting technique into a tool, called Dispatcher, to analyze botnet network communications and even inject new information into the communications stream. The researchers tested the approach on a complex botnet known as MegaD, which made headlines in early 2008 when security firms noticed it was responsible for nearly a third of spam traffic worldwide.