Skip to Content
Uncategorized

How Mechanical Turk is Broken

Why the world’s most famous outsourcing hub for tiny tasks is littered with spam and shoddy workmanship.

Amazon’s Mechanical Turk is important because in a perfect world, it could be the ultimate hub for the “human cloud” – an amorphous, pan-national, always-on pool of labor usable by corporations and individuals for tasks of any scale. It could provide ready employment for part-time workers in in any country on earth. It could be the ultimate in frictionless, free markets for labor, if you’re into that sort of thing.

Amazon founder and CEO Jeff Bezos
(cc) James Duncan Davidson

Recently, I told you how a fairly simple test of the abilities of Mechanical Turk revealed that the service is failing to deliver on its core promise – providing developers and businesses with access to a pool of humans who can perform tasks better than machines.

Panos Ipeirotis, a computer scientist who somehow finds himself at the Stern School of Business of New York University, explains on his blog how Mechanical Turk arrived at this sorry state of affairs. Being the sort who likes to use Mechanical Turk in his own work – the service is an increasingly popular platform for experimentation by academics – Iperotis has been chronicling what ails the service for at least a year. Its main problem, he contends, is that the site has become what economists call a Market for Lemons:

*A market for lemons is a market where the sellers cannot evaluate beforehand the quality of the goods that they are buying. So, if you have two types of products (say good workers and low quality workers) and cannot tell who is whom, the price that the buyer is willing to pay will be proportional to the average quality of the worker. So the offered price will be between the price of a good worker and a low quality worker.

This drives good workers out of the market alltogether, further lowering the payment per task and leading to a downward spiral of wages and expectations.

The ravages of this cycle have been profound: from a service that launched with much fanfare in the geek world in 2006, Mechanical Turk has become a site where 40 percent of all tasks are spam, contends Ipeirotis. Wages on the site average between two and three dollars an hour, despite the fact that for one quarter of participants, called Turkers, Mechanical Turk is their primary source of income. This has led Luis Von Ahn, internet-famous creator of reCaptcha, to wonder whether or not Mechanical Turk has become the digital equivalent of an unregulated sweatshop, where laws on minimum wage are flaunted.

Mechanical Turk is purposefully an anonymous labor market – there is no way for buyers of labor on the service to determine in what country their laborers reside, for example, and the site lacks a reputation system for workers, beyond the rate at which their completed tasks are accepted by requesters.

These features make the site more closely resemble something like a software-based cloud; the problem with that model is that humans are not mechanized slaves that consistently churn out work or are eliminated and reprogrammed, like software algorithms. Without transparency and accountability – the core ingredients of what economists call signaling – the site will continue to function primarily as an object lesson in the ways that poorly constructed markets fail.

Follow Mims on Twitter or contact him via email.

Keep Reading

Most Popular

Large language models can do jaw-dropping things. But nobody knows exactly why.

And that's a problem. Figuring it out is one of the biggest scientific puzzles of our time and a crucial step towards controlling more powerful future models.

How scientists traced a mysterious covid case back to six toilets

When wastewater surveillance turns into a hunt for a single infected individual, the ethics get tricky.

The problem with plug-in hybrids? Their drivers.

Plug-in hybrids are often sold as a transition to EVs, but new data from Europe shows we’re still underestimating the emissions they produce.

It’s time to retire the term “user”

The proliferation of AI means we need a new word.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.