Get in the Game
On the frontier of gesture recognition technology, there may be no better judge of a killer app than a four-year-old. That’s who I enlisted when I first hooked my PlayStation 2 up to an EyeToy-an unassuming device that’s shaping up to be the Pong of gesture interfaces. Intuitive, fun, and physical, it embodies the promise of gesture recognition wares. That promise is freedom-freedom from 14-button controllers, keyboards, mice, cables. “Everyone agrees that the keyboard isn’t necessarily the most optimal way to interface,” says senior analyst Joe Laszlo of Jupiter Research, a technology research firm based in New York City.
The EyeToy might be the first gesture recognition device to deliver a viable alternative to a keyboard or game controller. The hardware is a black-ribbed, rectangular digital camera about the size of a deck of cards. It plugs into the USB port on the front of the PlayStation. For about $50, you get the camera plus a CD of 12 games. Once the device is connected, you put it on top of your television set and angle it forward. The outline of a human body appears in the center of the screen, and you position yourself before the camera so that you fill it in.
“Four-year-old, come hither!” I say, helping my daughter to stand in the middle of the outlined shape. She waves at herself cautiously. “Where’s the game?” she asks. “You’re in it!” I respond. In front of her image on-screen floats a swarm of multicolored discs. To make a selection, she must wave her hand over the disc representing the game she’d like to play. The games are simple, almost like 21st-century versions of the old Atari classics Tennis and Combat. There’s a boxing game, a juggling game, a dancing game.
My daughter likes the sound of Wishi Washi, so Wishi Washi it is. Foamy bubbles cover the screen in front of her image. The object of the game is to “wipe” the screen clean to the strains of Dixieland jazz. Hesitantly at first, she waves her arm as if she were making a snow angel, and a corresponding blob of bubbles on-screen vanishes. The camera is capturing her movements, real time; before long she realizes she can use more than just her hands. By the end, she is jumping, leaning, kicking, flapping, using every physical motion she can muster to wipe away the foam. It’s not often you see a gamer sweat.
And plenty of gamers are sweating over the EyeToy. In the gaming business, sales of 500,000 constitute a hit. As of March, more than 500,000 EyeToys had been sold in the United States and more than two million in Europe.
Video from the tiny EyeToy camera is compressed and fed through the USB port. Once inside the PlayStation, the video is processed through “conceptual subtraction,” which compares the images in successive frames. The entire transaction uses less than 10 percent of the PlayStation 2’s processing power, leaving a hefty 90 percent to render the explosions, foam baths, and other graphics features of the games themselves.
In its current iteration, the EyeToy is limited to motion detection, but later versions will include more advanced features. Sony has already developed EyeToy software that can, for example, track different colors in an environment-even different faces. And it can deliver the sort of gesture recognition capabilities that would make, say, a Harry Potter video game truly come to life: draw a triangle with your wand and unleash a firestorm on-screen; draw a circle and turn your enemy into snow. “You’ll be able to cast a different kind of magic spell according to the shapes you draw in the air,” says Richard Marks, special-projects manager for research and development at Sony Computer Entertainment of America. EyeToy is his brainchild. Marks began working in the field of “computer vision”-technology that enables computers to perceive their surroundings-while developing cameras for underwater robots at the Monterey Bay Aquarium Research Institute in Moss Landing, CA. “I thought the PlayStation 2 would be good at computer vision,” he says.
But there are shortcomings. The USB port’s limited data-handling capacity results in fuzzy video and makes multiplayer online EyeToy experiences impossible. And the software can have difficulty discerning a player’s movements in a bright and busy environment-such as a typical family room. Marks says these problems will fade when the PlayStation 3 hits shelves, probably some time in 2006. The next-generation console will include a USB 2.0 port-as much as 40 times faster than a USB 1-which will reduce the fuzziness. Recognizing gestures against bright and busy backgrounds might require gamers to wave hot-pink wands or slip on gloves. Sony is distributing software tools that will help game developers exploit the new technology.
The ultimate goal is that you won’t need any prop at all. “The only thing you’ll need,” Marks says, “is your hand.”