Skip to Content
Uncategorized

How to Perfect Real-Time Crowdsourcing

The new techniques behind instant crowdsourcing makes human intelligence available on demand for the first time.

One of the great goals of computer science is to embed human-like intelligence in common applications like image processing, robotic control and so on. Until recently the focus has been to develop an artificial intelligence that can do these jobs. 

But there’s another option: using real humans via some kind of crowdsourcing process. One well known example involves the CAPTCHA test which can identify humans from machines by asking them to identify words so badly distorted that automated systems cannot read them. 

However, spammers are known to farm out these tasks to humans via crowdsourcing systems that pay in the region of 0.5 cents per 1000 words solved. 

Might not a similar process work for legitimate tasks such as building human intelligence into real world applications?

The problem, of course, is latency. Nobody wants to sit around for 20 minutes while a worker with the skills to steer your robotic waiter is crowdsourced from the other side of the world.

So how quickly can a crowd be put into action.?That’s the question tackled today by Michael Bernstein at the Massachusetts Institute of Technology in Cambridge and a few pals. 

In the past, these guys have found ways to bring a crowd to bear in about two seconds. That’s quick. But the reaction time is limited to how quickly a worker responds to an alert.

Now these guys say they’ve find a way to reduce the reaction time to 500 milliseconds–that’s effectively realtime. A system with a half second latency could turn crowdsourcing into a very different kind of resource. 

The idea that Bernstein and co have come up with is straightforward. These guys simply “precruit” a crowd and keep them on standby until a task becomes available. Effectively, they’re paying workers a retainer so that they are available immediately when needed

The difficulty is in the messy details of precruitment. How many workers do you need to keep on retainer, how do cope with drop outs  and how do you keep people interested so that they are available to work at a fraction of a second’s notice?

Bernstein and co have used an idea called queuing theory to work out how to optimise the process of precruitment according to how often the task comes up, how long it takes and so on. 

They’ve also developed an interesting psychological trick to keep workers ready for action. When workers are precruited, a screen opens up on their computer which downloads the task. The download occurs extremely quickly but if no task is to hand, the screen shows a “loading” bar. 

It turns out that the loading bar keeps workers focused on the forthcoming task for up to ten seconds, at which point their attention begins to wander. At that point, if no task materialises, the worker can be paid off.

Bernstein and co have even tested how well this works using a whack-a-mole type task which appears on workers screens after a randomly chosen period between 0 and 20 seconds. They recruited 50 workers to carry out 373 whacks and found the median length of time between the mole’s appearance and the worker moving the mouse toward the mole to click on it was 0.50 seconds.

“Our model suggests…that crowds could be recruited effectively instantaneously,” they say.

That could change the nature of crowdsourcing. Bernstein suggest that real time crowdsourcing could be used to point cameras, control robots and produce instant opinion polls.

But first crowdsourcers will have to change the way they do business. Bernstein and co suggest that they could build a retainer into their system design so that they have a pool of ready-to-go workers available at any instant. 

Workers could even be assessed at how good they are for this kind of task allowing them to build up a reputation for the work. Bernstein and co suggest two new reputation statistics–the percentage of time workers respond to a precruitment request and how quickly they respond on these occasions.

Shouldn’t be too hard to set up. An interesting new business model for the likes of Mechanical Turk and others or perhaps for an enterprising new start up.

Ref: arxiv.org/abs/1204.2995: Analytic Methods for Optimising Real time Crowdsourcing 

Keep Reading

Most Popular

Large language models can do jaw-dropping things. But nobody knows exactly why.

And that's a problem. Figuring it out is one of the biggest scientific puzzles of our time and a crucial step towards controlling more powerful future models.

OpenAI teases an amazing new generative video model called Sora

The firm is sharing Sora with a small group of safety testers but the rest of us will have to wait to learn more.

Google’s Gemini is now in everything. Here’s how you can try it out.

Gmail, Docs, and more will now come with Gemini baked in. But Europeans will have to wait before they can download the app.

This baby with a head camera helped teach an AI how kids learn language

A neural network trained on the experiences of a single young child managed to learn one of the core components of language: how to match words to the objects they represent.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.