Hello,

We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

Managing Users by the Million

At the heart of social networks are their enormous repositories of data: personal details and messages, huge quantities of photos and videos, and the complex web of connections that reflect the actual social relationships among users. Successful networks are those that have mastered how to store, secure, and quickly access and analyze this data.

Computing power: Featuring energy-saving ­innovations, ­Facebook’s new data center in ­Prineville, Oregon, was custom built to handle the social network’s demands.

The numbers are staggering. By late last year, Twitter’s users were generating 12 terabytes a day, which adds up to four peta­bytes a year, or the equivalent of 83,000 Blu-ray video disks—and that’s assuming no further user growth.

This story is part of our July/August 2011 Issue
See the rest of the issue
Subscribe

The networks rely partly on hardware to deal with this flood of data, building large data centers stuffed with servers. The other part of the solution is the software: many companies are contributing to open-source code designed to handle big databases. Twitter uses an open-source database called Cassandra that’s designed to work at large scales, with processing tasks distributed across a variety of relatively cheap servers.

In addition to storing data, keeping up with users is a challenge, even when they do something as apparently simple as clicking a “Like” button. Every time a user reports having watched a television program, for example, there’s already been “a lot of calculation to support that,” explains Alex Iskold, founder and CEO of Adaptive Blue, the company that maintains the entertainment-oriented social network GetGlue (see “Turn On, Check In). Each such note causes a cascade of calculations about what other media content to recommend, what information to display to other users, and whether any promotional incentives should be offered to the user. “The hardest thing to deal with is bursts,” Iskold says, referring to the surges in traffic during big events like the Oscars.

The information that social networks provide about users’ connections and interests has, despite initial doubts about its commercial value, proved incredibly lucrative. Social networks typically analyze the personal information provided by users to offer advertisers closely targeted commercial placement, a business that’s worth billions of dollars per year and growing. The value lies in “the combination of technology and identity,” says Jascha Kaykas-Wolff, senior vice president of marketing and customer success for Involver, a company that builds technology to help its customers create social-marketing campaigns. Data analysis tools are also being used to improve search results (see “Personalized Search).

The value of these networks attracts scammers as well as advertisers. In 2011, the security company Sophos reported that 40 percent of those who use social-network sites have received malware, 43 percent have been subjected to phishing attacks, and 67 percent have received spam. In response, network operators have begun watching for patterns that indicate malicious activity (for example, a link being shared among users faster than a human could reasonably accomplish it) and trying to develop technology to block these attacks before they reach users. The social-gaming network Zynga tracks sites that host hacks, bots, and cheats and monitors users suspected of bad behavior. Facebook has also recently introduced the option of texting a pass code to a user’s phone when that person’s account is accessed from a new computer, in hopes of preventing unauthorized access to an account if a password is compromised.

The AI revolution is here. Will you lead or follow?
Join us at EmTech Digital 2019.

Register now
Want more award-winning journalism? Subscribe to Insider Plus.
  • Insider Plus {! insider.prices.plus !}*

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus the digital magazine, extensive archive, ad-free web experience, and discounts to partner offerings and MIT Technology Review events.

    See details+

    Print + Digital Magazine (6 bi-monthly issues)

    Unlimited online access including all articles, multimedia, and more

    The Download newsletter with top tech stories delivered daily to your inbox

    Technology Review PDF magazine archive, including articles, images, and covers dating back to 1899

    10% Discount to MIT Technology Review events and MIT Press

    Ad-free website experience

/3
You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.