A new kind of botnet—a network of malware-infected PCs—behaves less like an army and more like a decentralized terrorist network, experts say. It can survive decapitation strikes, evade conventional defenses, and even wipe out competing criminal networks.

The botnet’s resilience is due to a super-sophisticated piece of malicious software known as TDL-4, which in the first three months of 2011 infected more than 4.5 million computers around the world, about a third of them in the United States.

The emergence of TDL-4 shows that the business of installing malicious code on PCs is thriving. Such code is used to conduct spam campaigns and various forms of theft and fraud, such as siphoning off passwords and other sensitive data. It’s also been used in the billion-dollar epidemic of fake anti-virus scams.

“Ultimately TDL-4 is simply a tool for maintaining and protecting a compromised platform for fraud,” says Eric Howes, malware analyst for GFI Software, a security company. “It’s part of the black service economy for malware, which has matured considerably over the past five years and which really needs a lot more light shed on it.”

Unlike other botnets, the TDL-4 network doesn’t rely on a few central “command-and-control” servers to pass along instructions and updates to all the infected computers. Instead, computers infected with TDL-4 pass along instructions to one another using public peer-to-peer networks. This makes it a “decentralized, server-less botnet,” wrote Sergey Golovanov, a malware researcher at the Moscow-based security company Kaspersky Lab, on this blog describing the new threat.

“The owners of TDL are essentially trying to create an ‘indestructible’ botnet that is protected against attacks, competitors, and antivirus companies,” Golovanov wrote. He added that it “is one of the most technologically sophisticated, and most complex-to-analyze malware.”

The TDL-4 botnet also breaks new ground by using an encryption algorithm that hides its communications from traffic-analysis tools. This is an apparent response to efforts by researchers to discover infected machines and disable botnets by monitoring their communication patterns, rather than simply identifying the presence of the malicious code.

Demonstrating that there is no honor among malicious software writers, TDL-4 scans for and deletes 20 of the most common forms of competing malware, so it can keep infected machines all to itself. “It’s interesting to mention that the features are generally oriented toward achieving perfect stealth, resilience, and getting rid of ‘competitor’ malware,” says Costin Raiu, another malware researcher at Kaspersky.

Distributed by criminal freelancers called affiliates, who get paid between $20 and $200 for every 1,000 infected machines, TDL-4 lurks on porn sites and some video and file-storage services, among other places, where it can be automatically installed using vulnerabilities in a victim’s browser or operating system.

Once TDL-4 infects a computer, it downloads and installs as many as 30 pieces of other malicious software—including spam-sending bots and password-stealing programs. “There are other malware-writing groups out there, but the gang behind [this one] is specifically targeted on delivering high-tech malware for profit,” says Raiu.