Computer malware can often evade antivirus security software if the author changes a few lines of code or designs the program to automatically mutate before each new infection.

Artificial neural networks, trained to recognize the characteristics of malicious code by looking at millions of examples of malware and non-malware files, could perhaps offer a far better way to catch such nefarious code. An approach known as deep learning, which involves training a network with many layers of simulated neurons using huge quantities of data, is being tested by several companies.

An Israeli startup called Deep Instinct plans to launch a security service based on the approach next week. The company claims that its software is significantly better at catching modified versions of existing malware than current antivirus software. Those claims have yet to be independently verified, but others are exploring the use of deep learning for antivirus software, and their published results suggest it could help turn the tide in the battle against malware infections.

Deep learning involves training a large network of simulated neurons and synapses to recognize abstract or complex patterns from example data. When fed a sufficiently large number of examples, such a network will correctly identify new examples that seem different on a basic level. A deep learning system can, for example, be trained to recognize a particular person’s face using thousands of images, and then spot that person in new photos, even ones taken in poor lighting or from an odd angle.

Eli David, cofounder and chief technology officer of Deep Instinct and a lecturer in machine learning at Bar-Ilan University in Israel, says his company trains its deep learning network using thousands of different parameters of different files. This time-consuming and computationally intensive process, which runs on a cluster of GPUs, is used to generate a static neural network that is then distributed to end users, David says. The network sent to users cannot be updated, which means it does not require as much computer power to run; but it can recognize, and flag, new malware.

According to the Virus Bulletin, an independent organization that tests security software, the best commercial antivirus can catch around 87 percent of all new threats several months after the software was last updated.

David says that in the company’s own tests, its software was able to detect 20 percent more new malware than existing antivirus software. Essentially it can tell if a file is similar enough to an existing piece of malware to make it suspicious. Existing antivirus software may be fooled if the particular string of code it is using for detection has been altered. “Deep learning is extremely resilient to noise,” he says. “That’s the idea here as well.”

A similar malware-detection deep learning network was developed by three researchers from Microsoft, together with George Dahl, who at the time was a student from one of the world’s leading deep learning labs at the University of Toronto, and is now a research scientist at Google.

A paper published by the researchers describes how they fed features including file strings and application programming interface parameters into several customized deep learning networks. After training the network using 2.6 million examples, the researchers write, the system was able to detect new instances of malware with “state-of-the-art performance.”

Another paper, published online by two researchers at the security company Invincea, describes a different effort to build a deep learning system for malware detection. The pair says that their deep learning system was able to detect new malware with a reliability of 95 percent and an error rate of .01 percent. 

It is unsurprising that deep learning is being considered for improving security software. Many large tech companies and startups are now pursuing deep learning aggressively. The approach has already improved the performance of handwriting recognition and voice recognition software; and it is increasingly being applied to much more complex tasks such as natural language understanding (see “Teaching Machines to Understand Us”).

George Cybenko, a professor at Dartmouth College who studies the use of machine learning in computer security, says that the idea of using neural networks to scan for malware goes back more than a decade. But he says the emergence of deep learning will probably cause companies to give the approach a closer look.

Cybenko says the performance being claimed for deep learning virus detection systems would be “a breakthrough,” although the results will have to be tested scientifically. He also notes that virus writers are notoriously persistent. “If there’s a breakthrough, they’re going to do some R&D and come up with a new approach.”