A Turing Test for Computer Game Bots

A new contest could help develop better AI for games and other applications.

Can a computer fool expert gamers into believing it's one of them? That was the question posed at the second annual BotPrize, a three-month contest that concluded today at the IEEE Computational Symposium on Intelligence and Games in Milan.

Prize fighter: A screenshot from Unreal Tournament 2004, the computer game used in the BotPrize competition. Credit: Epic Games

Multimedia   Watch a contestant in last year's competition.

The contest challenges programmers to create a software "bot" to control a game character that can pass for human, as judged by a panel of experts. The goal is not only to improve AI in entertainment, but also to fuel advances in non-gaming applications of AI. The BotPrize challenge is a variant of the Turing test, devised by Alan Turing, which challenges a machine to convince a panel of judges that it is a human in a text-only conversation.

"The BotPrize is important for AI in gaming because it aims to show how AI can make games more fun to play, by providing more interesting opponents for game players," says Philip Hingston, associate professor in the School of Computer and Information Science at Edith Cowan University in Perth, Australia, and an overseer of the competition. "It is also important for AI in general because it highlights a central question in AI: How is human intelligence related to computer intelligence?"

This year's BotPrize drew 15 entrants from Japan, the United Kingdom, the United States, Italy, Spain, Brazil, and Canada. Entrants created bots for Unreal Tournament 2004, a first-person shoot-'em-up in which gamers compete against each other for the most virtual kills. For the contest, in-game chatting was disabled so that bots could be evaluated for their so-called "humanness" by "physical" behavior alone. And, to elicit more spontaneity, contestants were given weapons that behaved differently from the ones ordinarily used in the game.

Story continues below

Each expert judge on the prize panel took turns shooting against two unidentified opponents-one human-controlled, the other a bot created by a contestant. After 10 to 15 minutes, the judge tried to identify the AI. To win the big prize, worth $6,000, a bot had to fool at least 80% of the judges. As in last year's competition, however, none of the participants was able to pull off this feat. A minor award worth $1,700, for the most "human-like" bot, was awarded to Jeremy Cathran, from the University of Southern California, for his entry, called sqlitebot.

Artificial intelligence has long been crucial to creating convincing and compelling computer games, whether a player is competing against drivers in Mario Kart on the Nintendo Wii or alien invaders in Halo 3 for the Microsoft's Xbox 360 games console. And, as competition increases in the $21 billion game industry, developers are striving to make game AI even more convincing. But creating a good bot presents a formidable challenge, says Steve Polge, lead programmer of Epic Games, the company that created Unreal Tournament. "You don't always want your AI to perform just like a human," he says. "Humans can be pretty annoying and obnoxious opponents." Instead, Polge says, developers often strive for "AI that can make unexpected plans and present emergent and surprising challenges to the player, which will definitely lead to better games."