Connectivity

The Surprising Light Side of the Dark Web

Indexing the dark Web offers a way to track crime—and shows that the hidden realm is a refuge for people who fear persecution.

Criminals selling Social Security numbers, the world’s largest social network, and transgender people seeking a safe place to talk are all part of the diverse world known as the dark Web—sites that can be accessed only using special software, usually intended to protect the identity and location of the websites and people using them.

The dark Web pierces the public consciousness only through incidents like the 2013 capture of Ross Ulbricht, who operated the Silk Road marketplace that hosted sales of illegal drugs and other contraband. But security companies and researchers are starting to deploy software that crawls the dark Web, similar to the way a conventional search engine indexes regular Web pages. As well as ferreting out or even preventing crime, these tools are showing that the dark Web has a significant light side.

Eric Michaud, founder and CEO of Darksum, has hundreds of computers indexing the contents of dark Web sites operated through the Tor anonymity network, which is believed to be home to its biggest and most active piece. He recently teamed up with U.K. security company Intelliagg to perform a kind of census.

The companies found that there are about 30,000 dark Web addresses online at any moment, although that includes many not serving up Web pages, such as chat servers, or configured to prevent casual browsing. Darksum and Intelliagg were able to use machine-learning software trained to classify a site's content on more than 13,000 sites. Just over half were tagged as hosting illegal activity of some kind. “That really surprised us,” says Michaud. “We thought it would be way worse.”

In addition to sites where people trade leaked data (about 28 percent of all sites logged), illegal pornography, or physical contraband, Michaud says, it was common to find forums used by communities looking for an ultraprivate space to socialize.

“These are people who want to meet others with like interests but not have those interests attached to their identity because it could be very bad for them,” says Michaud. “For example, there are trans forums where people go to talk about how they go about their daily lives.” Sites dedicated to the furry subculture also appear to be common, he says.

Some well-known companies have set up in the dark Web in recent years. Facebook launched a Tor hidden-services version of its site in 2014, saying that it would help people whose Internet activity is monitored or censored (see “Dark Web Version of Facebook Shows a New Way to Secure the Web”). Investigative news nonprofit ProPublica launched a hidden version of its own site in January, citing similar reasons.

The Darksum and Intelliagg findings tally with a recent study by two researchers at King’s College London published last month. Their software sorted 2,700 Tor hidden sites into categories for content, estimating that 57 percent hosted illegal material.

However, knowing how many hidden-services sites there are is impossible, says Staffan Truvé, chief technology officer at Recorded Future, a security company that tracks online information from sources including the dark Web to warn companies of impending trouble.

A search engine like Google can slurp up most of the Web by following the links between pages. “That doesn’t work on the dark Web because it’s not as interconnected,” says Truvé. Recorded Future and others discover pages by following dark Web links posted on conventional sites and consulting manually maintained directories, which are not exhaustive or up to date.

Still, Darksum and Recorded Future say their indexes are complete enough to help companies or law enforcement search the dark Web or get notified when specific content appears.

Michaud says his company recently helped one country’s national police force discover that officers’ data had been leaked. One company was able to neutralize a plan to cause a disturbance at an executive’s home by picking up mention of it on the dark Web, he says. Truvé says his company has been able to track the spread of certain kinds of malware from specialized forums in the dark Web to less sophisticated criminals who get their tools on the open Web.

 

 

Cut off? Read unlimited articles today.

Become an Insider
Already an Insider? Log in.

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

More from Connectivity

What it means to be constantly connected with each other and vast sources of information.

Want more award-winning journalism? Subscribe and become an Insider.
  • Insider Premium {! insider.prices.premium !}*

    {! insider.display.menuOptionsLabel !}

    Our award winning magazine, unlimited access to our story archive, special discounts to MIT Technology Review Events, and exclusive content.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

    First Look. Exclusive early access to stories.

    Insider Conversations. Listen in as our editors talk to innovators from around the world.

  • Insider Plus {! insider.prices.plus !}* Best Value

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus ad-free web experience, select discounts to partner offerings and MIT Technology Review events

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

  • Insider Basic {! insider.prices.basic !}*

    {! insider.display.menuOptionsLabel !}

    Six issues of our award winning magazine and daily delivery of The Download, our newsletter of what’s important in technology and innovation.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.