What AI Needs to Learn to Master Alien Warfare

“StarCraft is interesting for many reasons,” says Oriol Vinyals, the DeepMind researcher who is leading the project. The fact that players often get only a glimpse of their opponents’ activities, for instance, mean that algorithms will need to develop better ways of storing information in memory. “Memory is critical,” Vinyals says. “What you see now is not what you saw a while ago, and something specific that might have happened a minute ago might make you [want to] act differently.” 

DeepMind has built an impressive reputation on building AI programs capable of playing various types of games with superhuman skill. The company began by conquering various Atari games and more recently it took on the extremely complex and abstract board game Go (see “DeepMind’s AI Masters the Game of Go a Decade Earlier than Expected”).

To master these games DeepMind’s researchers used a machine-learning technique called reinforcement learning. Machine learning lets a computer figure out how to do something for itself, without requiring explicit instructions. Reinforcement learning, which is inspired by the way animals seem to learn, enables learning through experimentation with positive feedback (see “10 Breakthrough Technologies 2017: Reinforcement Learning”). However, Vinyals says applying reinforcement learning to StarCraft will be more difficult because it takes such a long time for each game to unfold. “An action I take now only has a consequence much later,” he says.

Within StarCraft, players compete as one of three races: the humanlike Terrans, the cyborg Protoss, or the insectoid Zerg. Battles involve complex strategic actions like mining resources and constructing bases, as well as protracted battle sequences. StarCraft is also the most popular spectator e-sport, and in South Korea especially, tournaments are often played in massive stadiums and shown live on television. Prominent players have welcomed the prospect of matching up against AI programs, but DeepMind hasn’t yet said when this might happen (see “StarCraft Pros Are Ready to Battle AI”).

The tools developed by DeepMind and Blizzard will make it much easier for AI researchers to deploy and test machine-learning algorithms inside StarCraft. The tools will provide AI agents with the same view of the game and interface that human players have. They also make it possible to limit the speed with which a program can execute its actions. This can ensure that a program has to rely on the same intellectual tools as a person does.

StarCraft has been used as a research platform for some time, but it has been relatively challenging to exploit. Vinyals, an expert StarCraft player himself, did pioneering work building bots for StarCraft as a student at the University of California, Berkeley (see “35 Innovators Under 35, 2016: Oriol Vinyals”). Teams at Facebook and the Chinese company Alibaba have also published StarCraft research. DeepMind is publishing a paper at a major machine-learning conference this week showing how existing algorithms perform with the game.

It’s possible that quite different approaches might be needed to master a game like StarCraft, however. Some other researchers have found success using approaches taken from game theory to make progress in other games of “imperfect information,” and where bluffing is important. Earlier this year, Tuomas Sandholm, a professor at Carnegie Mellon University, and one of his students, Noam Brown, built a program, called Libratus, that beat several professional players at heads up, or two-player, no-limit Texas hold’em. Libratus employed a very sophisticated algorithm to calculate the optimal strategy throughout a game (see “Why Poker Is a Big Deal for Artificial Intelligence”). And coincidentally, Brown has been interning at DeepMind this summer.

Jacob Repp, a principle engineer at Blizzard, says his company is interested in seeing if sophisticated AI agents could make the game more interesting, either by playing against people or collaborating with them. It is already possible to create agents in the game that follow scripted commands. Repp says it would be interesting to have those agents use machine learning to some degree, too. And he says the company is exploring these sorts of ideas. “We’re finding that these tools are very useful for the process of making games and designing features in games,” he says.