Hello,

We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

A View from Henry Lieberman

Watson on Jeopardy, Part 3

IBM’s machine improves its performance and turns the game into a rout.

  • February 16, 2011

The second night of the IBM/Jeopardy match was a rout for the Watson program. The questions in this episode were harder, and, in my view, the match was more interesting.

Watson performed admirably. As I explained in my previous essays, the real game isn’t knowing facts; it’s the ability to do enough natural-language understanding to figure out what the question is asking for.

Clue: An étude is a composition that … The name is French for this

Here, it has to figure out that what’s being asked for is the English translation of the French word étude. Then do the translation. It got it. Bravo!

Even in its mistakes, I was impressed. And looking at mistakes that people make, too, tells you a lot. In one case, both Watson and the humans had trouble interpreting the question.

Clue: In May 2010, five paintings … left Paris’s art museum of this period.

Watson: Picasso (Wrong)

Ken Jennings: Cubism (Wrong)

Brad Rutter: Impressionism (Wrong)

Correct answer: Modern Art.

The question isn’t asking for the name of an artist or an art period—it’s asking for the name of a museum! (Musée d’Art Moderne—Museum of Modern Art). There is a Picasso Museum in Paris, but there is no Museum of Cubism nor Museum of Impressionism. Modern Art was Watson’s third choice. Kudos to Watson for at least getting that far!

The Final Jeopardy question built up the suspense.

Category: US Cities

Clue: Its largest airport was named after a World War II hero; its second largest for a World War II battle.

Human contestants’ answer: Chicago (Correct) Watson’s answer: Toronto (Wrong)

The question was interesting because it requires doing an intersection—figuring out which items several sets have in common. You think of every airport you know. Which ones are named after World War II heroes? Probably you can’t remember too many war heroes, but you do know which airports are named after people. Which cities have two airports? New York? Kennedy a hero, sure, but (I’m a New Yorker) LaGuardia was a mayor, not a battle. And so on. Cognitive science experiments show that people aren’t very good at doing intersections (What city is named after an animal? Answer: Buffalo).

I don’t know the names of the Toronto airports, but the answer is sure to antagonize Canadians, since the category was US Cities. I don’t know why Watson made that mistake. But it illustrates the problem of

overconstrained situations—you can’t find any really good answer, and you’ve got to break the rules. A better choice would have been to stay in the US and take the risk that you would be wrong about LaGuardia.

But Watson did have some game strategy that saved the day for it. It realized it was ahead and it couldn’t be caught by the other contestants, so it didn’t bet much on the final outcome. If it figured that out, that’s more impressive than knowing the facts of the answer. Strategy sometimes trumps knowledge.

Henry Lieberman is a research scientist who works on artificial intelligence at the Media Laboratory at MIT.

Couldn't make it to EmTech Next to meet experts in AI, Robotics and the Economy?

Go behind the scenes and check out our video
More from Intelligent Machines

Artificial intelligence and robots are transforming how we work and live.

Want more award-winning journalism? Subscribe and become an Insider.
  • Insider Plus {! insider.prices.plus !}* Best Value

    {! insider.display.menuOptionsLabel !}

    Everything included in Insider Basic, plus the digital magazine, extensive archive, ad-free web experience, and discounts to partner offerings and MIT Technology Review events.

    See details+

    Print + Digital Magazine (6 bi-monthly issues)

    Unlimited online access including all articles, multimedia, and more

    The Download newsletter with top tech stories delivered daily to your inbox

    Technology Review PDF magazine archive, including articles, images, and covers dating back to 1899

    10% Discount to MIT Technology Review events and MIT Press

    Ad-free website experience

  • Insider Basic {! insider.prices.basic !}*

    {! insider.display.menuOptionsLabel !}

    Six issues of our award winning print magazine, unlimited online access plus The Download with the top tech stories delivered daily to your inbox.

    See details+

    Print Magazine (6 bi-monthly issues)

    Unlimited online access including all articles, multimedia, and more

    The Download newsletter with top tech stories delivered daily to your inbox

  • Insider Online Only {! insider.prices.online !}*

    {! insider.display.menuOptionsLabel !}

    Unlimited online access including articles and video, plus The Download with the top tech stories delivered daily to your inbox.

    See details+

    Unlimited online access including all articles, multimedia, and more

    The Download newsletter with top tech stories delivered daily to your inbox

/3
You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.