How Mechanical Turk is Broken

Amazon’s Mechanical Turk is important because in a perfect world, it could be the ultimate hub for the “human cloud” – an amorphous, pan-national, always-on pool of labor usable by corporations and individuals for tasks of any scale. It could provide ready employment for part-time workers in in any country on earth. It could be the ultimate in frictionless, free markets for labor, if you’re into that sort of thing.

Recently, I told you how a fairly simple test of the abilities of Mechanical Turk revealed that the service is failing to deliver on its core promise – providing developers and businesses with access to a pool of humans who can perform tasks better than machines.

Panos Ipeirotis, a computer scientist who somehow finds himself at the Stern School of Business of New York University, explains on his blog how Mechanical Turk arrived at this sorry state of affairs. Being the sort who likes to use Mechanical Turk in his own work – the service is an increasingly popular platform for experimentation by academics – Iperotis has been chronicling what ails the service for at least a year. Its main problem, he contends, is that the site has become what economists call a Market for Lemons:

*A market for lemons is a market where the sellers cannot evaluate beforehand the quality of the goods that they are buying. So, if you have two types of products (say good workers and low quality workers) and cannot tell who is whom, the price that the buyer is willing to pay will be proportional to the average quality of the worker. So the offered price will be between the price of a good worker and a low quality worker.

This drives good workers out of the market alltogether, further lowering the payment per task and leading to a downward spiral of wages and expectations.

The ravages of this cycle have been profound: from a service that launched with much fanfare in the geek world in 2006, Mechanical Turk has become a site where 40 percent of all tasks are spam, contends Ipeirotis. Wages on the site average between two and three dollars an hour, despite the fact that for one quarter of participants, called Turkers, Mechanical Turk is their primary source of income. This has led Luis Von Ahn, internet-famous creator of reCaptcha, to wonder whether or not Mechanical Turk has become the digital equivalent of an unregulated sweatshop, where laws on minimum wage are flaunted.

Mechanical Turk is purposefully an anonymous labor market – there is no way for buyers of labor on the service to determine in what country their laborers reside, for example, and the site lacks a reputation system for workers, beyond the rate at which their completed tasks are accepted by requesters.

These features make the site more closely resemble something like a software-based cloud; the problem with that model is that humans are not mechanized slaves that consistently churn out work or are eliminated and reprogrammed, like software algorithms. Without transparency and accountability – the core ingredients of what economists call signaling – the site will continue to function primarily as an object lesson in the ways that poorly constructed markets fail.

Follow Mims on Twitter or contact him via email.