Skip to Content
Artificial intelligence

AI Beats Humans at Reading Comprehension, but It Still Doesn’t Truly Comprehend Language

January 15, 2018

While Alibaba and Microsoft may have developed AIs that outperform humans at a comprehension test, there are still tough natural language challenges facing machines.

The challenge: The Stanford University quiz, based on 500 Wikipedia articles, tests comprehension of words and sentences with questions like: “Which group headlined the Super Bowl 50 half-time show?”

The scores: Humans get 82.304. Alibaba’s AI achieved 82.44, Microsoft’s 82.650.

What they say: Alibaba chief scientist Luo Si says it “means objective questions such as ‘what causes rain’ can now be answered with high accuracy by machines,” and plans to use the technology in real-world applications like customer service.

But: This isn’t comprehension the way humans think of it. It’s neat, but the AI doesn’t really understand what it reads—it doesn’t know what “British rock group Coldplay” really is, besides it being the answer to the Super Bowl question. And there are far harder language problems that humans still beat computers at.

Deep Dive

Artificial intelligence

A Roomba recorded a woman on the toilet. How did screenshots end up on Facebook?

Robot vacuum companies say your images are safe, but a sprawling global supply chain for data from our devices creates risk.

The viral AI avatar app Lensa undressed me—without my consent

My avatars were cartoonishly pornified, while my male colleagues got to be astronauts, explorers, and inventors.

Roomba testers feel misled after intimate images ended up on Facebook

An MIT Technology Review investigation recently revealed how images of a minor and a tester on the toilet ended up on social media. iRobot said it had consent to collect this kind of data from inside homes—but participants say otherwise.

How to spot AI-generated text

The internet is increasingly awash with text written by AI software. We need new tools to detect it.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.