Skip to Content
Artificial intelligence

Researchers are struggling to replicate AI studies

February 16, 2018

Missing code and data are making it difficult to compare machine-learning work—and that may be hurting progress.

The problem:  Science reports that from a sample of 400 papers at top AI conferences in recent years, only 6 percent of presenters shared code. Just a third shared data, and a little over half shared summaries of their algorithms, known as pseudocode.

Why it matters: Without access to that information, it’s hard to reproduce a study’s findings. That makes it all but impossible to benchmark newly developed tools against existing ones, so it’s hard for researchers to know which direction to push future research.

How to solve it: Sometimes a lack of sharing may be understandable—say, if intellectual property is owned by a private firm. But there seems to be a wider-spread culture of keeping details under wraps. Some meetings and journals are now encouraging sharing; perhaps more ought to follow.

Deep Dive

Artificial intelligence

open sourcing language models concept
open sourcing language models concept

Meta has built a massive new language AI—and it’s giving it away for free

Facebook’s parent company is inviting researchers to pore over and pick apart the flaws in its version of GPT-3

Yann LeCun
Yann LeCun

Yann LeCun has a bold new vision for the future of AI

One of the godfathers of deep learning pulls together old ideas to sketch out a fresh path for AI, but raises as many questions as he answers.

images created by Google Imagen
images created by Google Imagen

The dark secret behind those cute AI-generated animal images

Google Brain has revealed its own image-making AI, called Imagen. But don't expect to see anything that isn't wholesome.

AGI is just chatter for now concept
AGI is just chatter for now concept

The hype around DeepMind’s new AI model misses what’s actually cool about it

Some worry that the chatter about these tools is doing the whole field a disservice.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.