Skip to Content

The Surprising Light Side of the Dark Web

Indexing the dark Web offers a way to track crime—and shows that the hidden realm is a refuge for people who fear persecution.
March 18, 2016

Criminals selling Social Security numbers, the world’s largest social network, and transgender people seeking a safe place to talk are all part of the diverse world known as the dark Web—sites that can be accessed only using special software, usually intended to protect the identity and location of the websites and people using them.

The dark Web pierces the public consciousness only through incidents like the 2013 capture of Ross Ulbricht, who operated the Silk Road marketplace that hosted sales of illegal drugs and other contraband. But security companies and researchers are starting to deploy software that crawls the dark Web, similar to the way a conventional search engine indexes regular Web pages. As well as ferreting out or even preventing crime, these tools are showing that the dark Web has a significant light side.

Eric Michaud, founder and CEO of Darksum, has hundreds of computers indexing the contents of dark Web sites operated through the Tor anonymity network, which is believed to be home to its biggest and most active piece. He recently teamed up with U.K. security company Intelliagg to perform a kind of census.

The companies found that there are about 30,000 dark Web addresses online at any moment, although that includes many not serving up Web pages, such as chat servers, or configured to prevent casual browsing. Darksum and Intelliagg were able to use machine-learning software trained to classify a site's content on more than 13,000 sites. Just over half were tagged as hosting illegal activity of some kind. “That really surprised us,” says Michaud. “We thought it would be way worse.”

In addition to sites where people trade leaked data (about 28 percent of all sites logged), illegal pornography, or physical contraband, Michaud says, it was common to find forums used by communities looking for an ultraprivate space to socialize.

“These are people who want to meet others with like interests but not have those interests attached to their identity because it could be very bad for them,” says Michaud. “For example, there are trans forums where people go to talk about how they go about their daily lives.” Sites dedicated to the furry subculture also appear to be common, he says.

Some well-known companies have set up in the dark Web in recent years. Facebook launched a Tor hidden-services version of its site in 2014, saying that it would help people whose Internet activity is monitored or censored (see “Dark Web Version of Facebook Shows a New Way to Secure the Web”). Investigative news nonprofit ProPublica launched a hidden version of its own site in January, citing similar reasons.

The Darksum and Intelliagg findings tally with a recent study by two researchers at King’s College London published last month. Their software sorted 2,700 Tor hidden sites into categories for content, estimating that 57 percent hosted illegal material.

However, knowing how many hidden-services sites there are is impossible, says Staffan Truvé, chief technology officer at Recorded Future, a security company that tracks online information from sources including the dark Web to warn companies of impending trouble.

A search engine like Google can slurp up most of the Web by following the links between pages. “That doesn’t work on the dark Web because it’s not as interconnected,” says Truvé. Recorded Future and others discover pages by following dark Web links posted on conventional sites and consulting manually maintained directories, which are not exhaustive or up to date.

Still, Darksum and Recorded Future say their indexes are complete enough to help companies or law enforcement search the dark Web or get notified when specific content appears.

Michaud says his company recently helped one country’s national police force discover that officers’ data had been leaked. One company was able to neutralize a plan to cause a disturbance at an executive’s home by picking up mention of it on the dark Web, he says. Truvé says his company has been able to track the spread of certain kinds of malware from specialized forums in the dark Web to less sophisticated criminals who get their tools on the open Web.

 

 

Keep Reading

Most Popular

A Roomba recorded a woman on the toilet. How did screenshots end up on Facebook?

Robot vacuum companies say your images are safe, but a sprawling global supply chain for data from our devices creates risk.

A startup says it’s begun releasing particles into the atmosphere, in an effort to tweak the climate

Make Sunsets is already attempting to earn revenue for geoengineering, a move likely to provoke widespread criticism.

10 Breakthrough Technologies 2023

Every year, we pick the 10 technologies that matter the most right now. We look for advances that will have a big impact on our lives and break down why they matter.

These exclusive satellite images show that Saudi Arabia’s sci-fi megacity is well underway

Weirdly, any recent work on The Line doesn’t show up on Google Maps. But we got the images anyway.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.