A new prototype can transform a notebook into a notebook computer, a wall into an interactive display, and the palm of your hand into a smart phone display. In fact, researchers at Microsoft and Carnegie Mellon University say their new shoulder-mounted device, called OmniTouch, can turn any nearby surface into an ad hoc interactive touch screen.

OmniTouch works by bringing together a miniature projector and an infrared depth camera, similar to the kind used in Microsoft's Kinect game console, to create a shoulder-worn system designed to interface with mobile devices such as smart phones, says co-inventor Chris Harrison, a postgraduate researcher at Carnegie Mellon's Human-Computer Interaction Institute in Pittsburgh and a former intern at Microsoft Research. Instead of relying on screens, buttons, or keys, the system monitors the user's environment for any available surfaces and projects an interactive display onto one or more of them.

OmniTouch does this automatically, using the depth information provided by the camera to build a 3-D model of the environment, says Harrison. The camera acquires depth information about the scene by emitting a patterned beam of infrared light and using the reflections to calculate where surfaces are in the room. This eliminates the need for external calibration markers. The system rebuilds the model dynamically as the user or the surface moves—for example, the position of a hand or the angle or orientation of a book—so the size, shape, and position of these projections match those of the improvised display surfaces, he says. OmniTouch "figures out what's in front you and fits everything on to it."

The system also monitors the environment for anything cylindrical and roughly finger-sized to work out when the user is interacting with it, again using depth information to determine when a finger or fingers make contact with a surface. This lets users interact with arbitrary surfaces just as they would a touch screen, says Harrison. Similarly, objects and icons on the ad hoc "screens" can be swiped and pinched to scroll and zoom, much like on a traditional touch screen. In one demonstration art application, for example, OmniTouch used a nearby wall or table as a canvas and the palm of the user's hand as the color palette.

The shoulder-mounted setup is completely impractical, admits Hrvoje Benko, a researcher in Natural Interaction Research group at Microsoft Research in Redmond, Washington, who also worked on the project, along with colleague Andrew Wilson. "But it's not where you mount it that counts," he says. "The core motivation was to push this idea of turning any available surface into an interactive surface." All the components used in OmniTouch are off the shelf and shrinking all the time. "So I don't think we're so far from it being made into a pendant or attached to glasses," says Benko.

Duncan Brumby, a researcher at the University College London Interaction Center, in England, calls OmniTouch a fun and novel form of interaction. The screen sizes of mobile devices can be limiting, he says. "There's a growing interest in this area of having ubiquitous, intangible displays embedded in the environment," he says. And although new generations of smart phones tend to have increasingly higher-quality displays, Brumby reckons users would be willing to put up with lower-quality projected images, given the right applications.