Google’s Experimental Smartphone Captures a Future Mapped in 3-D

Four months after Google unveiled Project Tango—a prototype Android smartphone with cameras and sensors that capture the phone’s environment in 3-D—developers are using the device to make cheap drones for surveying zones, more immersive video games, and even a body-part scanning system that could lead to a better-fitting suit.

As well as a normal front-facing camera and sensors that measure orientation and motion, the device has a fish-eye camera on its rear that sees 180 degrees and a depth-sensing camera that uses infrared to capture a 3-D representation of the surrounding world. So far, 200 of the phones have been handed out to researchers and companies, who are doing things like strapping them to drones and using them to scan and model body parts.

Researchers and developers experimenting with Project Tango say it could make it easier, quicker, and cheaper to scan and map all kinds of places and objects in three dimensions. Vijay Kumar, a University of Pennsylvania professor who studies autonomous robots, says that Tango cuts the cost of making a flying robot, and is well-suited to performing both sensing and control because it’s so lightweight and compact. He’s experimenting with one strapped to an autonomous quadcopter; the device allows the copter to continuously return to a preprogrammed midair spot even after being pushed and pulled away from it.

Kumar says getting the Tango device to work with his robot was relatively easy, taking several hours of programming, rather than several weeks. In part, this is because smartphone processors are good enough to handle the data robots need to operate, thus avoiding the need to hand the data off to another computer, he says.

Kumar’s goal for Tango is for the handset and robots to one day be made cheaply enough that he could buy 100 of them and send the swarm into a disaster site, where they would collectively build a 3-D interior map within a few seconds.

“That would be cool,” he says.

Closer to earth, Amir Rubin, CEO and cofounder of 3-D indoor map startup Paracosm, is considering entertainment-oriented possibilities for Tango: he’s working on a video game demo.

Paracosm’s demo involves using the handset to scan a room that you can then “play” as a level in the game. You move through that same room—in reality—to control the first-person gameplay on the device’s screen (eventually you will control a character within the game, he says).

Johnny Lee, Project Tango’s tech lead, wouldn’t say when devices using the technology could hit the market. And since the existing Tango phones are just prototypes, there are many issues to work out before you’d want to buy one. For example, while developers praise its ability to combine a number of features—3-D scanning capabilities, position and orientation sensors—into a smartphone-sized package, they also say its battery life is relatively short (though they decline to say how long it lasts, exactly); it can’t do much graphics processing; and it can overheat and shut down.

Kumar says he can’t run the Tango-equipped drone for more than five minutes at a time without risking it shutting down midflight. “That would be like a bird flying and then you suddenly chop off its brains and it’s going to crash,” he says.

Developers expect improvements over time.

“It’s still remarkably good,” says Raj Sareen, CEO and cofounder of Styku, a 3-D body-scanning startup. His company is working on a demo that uses its scanning software and Project Tango to create 3-D models of people’s heads—meant to show how effectively it can scan body parts.

To use the demo, you hold a Tango phone in front of someone’s face and move it in a half circle around them. Once this is done, the data is uploaded to a Styku server that builds a 3-D model from the depth data that is then downloaded to the phone (the current version of the phone isn’t powerful enough to build the model itself; but Sareen expects this to change in the next iteration).

Styku currently uses other scanning systems, including Microsoft’s Kinect. The technology, which is used by stores like Nordstrom to help customers find the best-fitting clothes based on precise body measurements, costs about $5,000 to $6,000. If 3-D depth sensors are built into phones, he says, people could just scan themselves using a Styku app—making it a lot easier to buy custom-made clothing online. Or people might track their health and fitness by comparing the measurements of different body parts over time.

“I think we’re still a few steps removed from that,” he says. “But we strongly believe that’s the future.”