The Surprising Light Side of the Dark Web

Criminals selling Social Security numbers, the world’s largest social network, and transgender people seeking a safe place to talk are all part of the diverse world known as the dark Web—sites that can be accessed only using special software, usually intended to protect the identity and location of the websites and people using them.

The dark Web pierces the public consciousness only through incidents like the 2013 capture of Ross Ulbricht, who operated the Silk Road marketplace that hosted sales of illegal drugs and other contraband. But security companies and researchers are starting to deploy software that crawls the dark Web, similar to the way a conventional search engine indexes regular Web pages. As well as ferreting out or even preventing crime, these tools are showing that the dark Web has a significant light side.

Eric Michaud, founder and CEO of Darksum, has hundreds of computers indexing the contents of dark Web sites operated through the Tor anonymity network, which is believed to be home to its biggest and most active piece. He recently teamed up with U.K. security company Intelliagg to perform a kind of census.

The companies found that there are about 30,000 dark Web addresses online at any moment, although that includes many not serving up Web pages, such as chat servers, or configured to prevent casual browsing. Darksum and Intelliagg were able to use machine-learning software trained to classify a site's content on more than 13,000 sites. Just over half were tagged as hosting illegal activity of some kind. “That really surprised us,” says Michaud. “We thought it would be way worse.”

In addition to sites where people trade leaked data (about 28 percent of all sites logged), illegal pornography, or physical contraband, Michaud says, it was common to find forums used by communities looking for an ultraprivate space to socialize.

“These are people who want to meet others with like interests but not have those interests attached to their identity because it could be very bad for them,” says Michaud. “For example, there are trans forums where people go to talk about how they go about their daily lives.” Sites dedicated to the furry subculture also appear to be common, he says.

Some well-known companies have set up in the dark Web in recent years. Facebook launched a Tor hidden-services version of its site in 2014, saying that it would help people whose Internet activity is monitored or censored (see “Dark Web Version of Facebook Shows a New Way to Secure the Web”). Investigative news nonprofit ProPublica launched a hidden version of its own site in January, citing similar reasons.

The Darksum and Intelliagg findings tally with a recent study by two researchers at King’s College London published last month. Their software sorted 2,700 Tor hidden sites into categories for content, estimating that 57 percent hosted illegal material.

However, knowing how many hidden-services sites there are is impossible, says Staffan Truvé, chief technology officer at Recorded Future, a security company that tracks online information from sources including the dark Web to warn companies of impending trouble.

A search engine like Google can slurp up most of the Web by following the links between pages. “That doesn’t work on the dark Web because it’s not as interconnected,” says Truvé. Recorded Future and others discover pages by following dark Web links posted on conventional sites and consulting manually maintained directories, which are not exhaustive or up to date.

Still, Darksum and Recorded Future say their indexes are complete enough to help companies or law enforcement search the dark Web or get notified when specific content appears.

Michaud says his company recently helped one country’s national police force discover that officers’ data had been leaked. One company was able to neutralize a plan to cause a disturbance at an executive’s home by picking up mention of it on the dark Web, he says. Truvé says his company has been able to track the spread of certain kinds of malware from specialized forums in the dark Web to less sophisticated criminals who get their tools on the open Web.