Christopher Mims

A View from Christopher Mims

How Mechanical Turk is Broken

Why the world’s most famous outsourcing hub for tiny tasks is littered with spam and shoddy workmanship.

  • January 3, 2010

Amazon’s Mechanical Turk is important because in a perfect world, it could be the ultimate hub for the “human cloud” – an amorphous, pan-national, always-on pool of labor usable by corporations and individuals for tasks of any scale. It could provide ready employment for part-time workers in in any country on earth. It could be the ultimate in frictionless, free markets for labor, if you’re into that sort of thing.

Amazon founder and CEO Jeff Bezos
(cc) James Duncan Davidson

Recently, I told you how a fairly simple test of the abilities of Mechanical Turk revealed that the service is failing to deliver on its core promise – providing developers and businesses with access to a pool of humans who can perform tasks better than machines.

Panos Ipeirotis, a computer scientist who somehow finds himself at the Stern School of Business of New York University, explains on his blog how Mechanical Turk arrived at this sorry state of affairs. Being the sort who likes to use Mechanical Turk in his own work – the service is an increasingly popular platform for experimentation by academics – Iperotis has been chronicling what ails the service for at least a year. Its main problem, he contends, is that the site has become what economists call a Market for Lemons:

*A market for lemons is a market where the sellers cannot evaluate beforehand the quality of the goods that they are buying. So, if you have two types of products (say good workers and low quality workers) and cannot tell who is whom, the price that the buyer is willing to pay will be proportional to the average quality of the worker. So the offered price will be between the price of a good worker and a low quality worker.

This drives good workers out of the market alltogether, further lowering the payment per task and leading to a downward spiral of wages and expectations.

The ravages of this cycle have been profound: from a service that launched with much fanfare in the geek world in 2006, Mechanical Turk has become a site where 40 percent of all tasks are spam, contends Ipeirotis. Wages on the site average between two and three dollars an hour, despite the fact that for one quarter of participants, called Turkers, Mechanical Turk is their primary source of income. This has led Luis Von Ahn, internet-famous creator of reCaptcha, to wonder whether or not Mechanical Turk has become the digital equivalent of an unregulated sweatshop, where laws on minimum wage are flaunted.

Mechanical Turk is purposefully an anonymous labor market – there is no way for buyers of labor on the service to determine in what country their laborers reside, for example, and the site lacks a reputation system for workers, beyond the rate at which their completed tasks are accepted by requesters.

These features make the site more closely resemble something like a software-based cloud; the problem with that model is that humans are not mechanized slaves that consistently churn out work or are eliminated and reprogrammed, like software algorithms. Without transparency and accountability – the core ingredients of what economists call signaling – the site will continue to function primarily as an object lesson in the ways that poorly constructed markets fail.

Follow Mims on Twitter or contact him via email.

Tech Obsessive?
Become an Insider to get the story behind the story — and before anyone else.

Subscribe today

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

Want more award-winning journalism? Subscribe to Insider Plus.
  • Insider Plus {! insider.prices.plus !}*

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus ad-free web experience, select discounts to partner offerings and MIT Technology Review events

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.