We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

Christopher Mims

A View from Christopher Mims

How Mechanical Turk is Broken

Why the world’s most famous outsourcing hub for tiny tasks is littered with spam and shoddy workmanship.

  • January 3, 2010

Amazon’s Mechanical Turk is important because in a perfect world, it could be the ultimate hub for the “human cloud” – an amorphous, pan-national, always-on pool of labor usable by corporations and individuals for tasks of any scale. It could provide ready employment for part-time workers in in any country on earth. It could be the ultimate in frictionless, free markets for labor, if you’re into that sort of thing.

Amazon founder and CEO Jeff Bezos
(cc) James Duncan Davidson

Recently, I told you how a fairly simple test of the abilities of Mechanical Turk revealed that the service is failing to deliver on its core promise – providing developers and businesses with access to a pool of humans who can perform tasks better than machines.

Panos Ipeirotis, a computer scientist who somehow finds himself at the Stern School of Business of New York University, explains on his blog how Mechanical Turk arrived at this sorry state of affairs. Being the sort who likes to use Mechanical Turk in his own work – the service is an increasingly popular platform for experimentation by academics – Iperotis has been chronicling what ails the service for at least a year. Its main problem, he contends, is that the site has become what economists call a Market for Lemons:

*A market for lemons is a market where the sellers cannot evaluate beforehand the quality of the goods that they are buying. So, if you have two types of products (say good workers and low quality workers) and cannot tell who is whom, the price that the buyer is willing to pay will be proportional to the average quality of the worker. So the offered price will be between the price of a good worker and a low quality worker.

This drives good workers out of the market alltogether, further lowering the payment per task and leading to a downward spiral of wages and expectations.

The ravages of this cycle have been profound: from a service that launched with much fanfare in the geek world in 2006, Mechanical Turk has become a site where 40 percent of all tasks are spam, contends Ipeirotis. Wages on the site average between two and three dollars an hour, despite the fact that for one quarter of participants, called Turkers, Mechanical Turk is their primary source of income. This has led Luis Von Ahn, internet-famous creator of reCaptcha, to wonder whether or not Mechanical Turk has become the digital equivalent of an unregulated sweatshop, where laws on minimum wage are flaunted.

Mechanical Turk is purposefully an anonymous labor market – there is no way for buyers of labor on the service to determine in what country their laborers reside, for example, and the site lacks a reputation system for workers, beyond the rate at which their completed tasks are accepted by requesters.

These features make the site more closely resemble something like a software-based cloud; the problem with that model is that humans are not mechanized slaves that consistently churn out work or are eliminated and reprogrammed, like software algorithms. Without transparency and accountability – the core ingredients of what economists call signaling – the site will continue to function primarily as an object lesson in the ways that poorly constructed markets fail.

Follow Mims on Twitter or contact him via email.

Cut off? Read unlimited articles today.

Become an Insider
Already an Insider? Log in.
Want more award-winning journalism? Subscribe to Insider Online Only.
  • Insider Online Only {! insider.prices.online !}*

    {! insider.display.menuOptionsLabel !}

    Unlimited online access including articles and video, plus The Download with the top tech stories delivered daily to your inbox.

    See details+

    Unlimited online access including all articles, multimedia, and more

    The Download newsletter with top tech stories delivered daily to your inbox

You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.