Augmented or mixed reality, which renders virtual images in a view of the real world, can be spectacular to experience. But it may be even more fun when you bring a friend.
Researchers in the Microsoft lab of Jaron Lanier are experimenting with multi-person augmented reality, and the results of their work could help shape the way the technology is commercialized and used. Lanier was a pioneer of virtual and augmented, or mixed, reality in the 1980s.
Microsoft is testing a commercial augmented reality product, called HoloLens. Lanier stresses that his work is separate from HoloLens and does not reflect how that product will develop. Still, multi-person mixed reality is a long-standing challenge for those interested in the technology. Beyond gaming, there is hope that virtual and augmented reality could prove useful for communications, collaboration, and for new ways of accessing and handling information.
Lanier’s project is called Comradre (and pronounced “comradery”). A video produced by Lanier’s lab shows several projects developed by student interns in which more than one person interacts with the same virtual object or phenomena. These headsets were made using smartphones and laptop computers, and the tracking of a person’s head movements are performed using external sensors. And most of the footage was shot through one of the headsets.
The video shows a number of experimental applications developed for the platform. Andrea Stevenson Won of Stanford University is testing the social implications of the technology with a system that highlights physical interactions with virtual effects. Judith Amores and Xavie Benavides from the MIT Media Lab created a way for real-world blocks to correspond with virtual objects and animations, offering a way for children to play in mixed reality, even if not everyone is wearing a headset.
Other students in the group created more utilitarian uses. Andrzej Banburski of the Perimeter Institute for Advanced Physics, for example, developed a tool for visualizing mathematical equations. And Kishore Rathinavel, from the University of North Carolina, Chapel Hill, created a system that allows sound waves to be detected and then shown visually. In one scene from the group’s video, the sound produced by Lanier’s flute playing is shown in mixed reality. Two other students, Victor Mateevitsi, at the University of Illinois at Chicago, and Gheric Speiginer of Georgia Tech, were also part of the group.
Both augmented reality and virtual reality—which provides a completely immersive experience—are currently hot areas of research and commercial development at several major technology companies. Facebook, Google, and Sony are all developing headsets and software for virtual reality. Meanwhile, a company called Magic Leap has raised more than $500 million in funding to develop a convincing form of augmented reality using holographic technology (see “Breakthrough Technologies 2015: Magic Leap”).
Microsoft demonstrated HoloLens last week at an event held for the launch of Windows 10, the latest version of its computer operating system. The demo showed an experimental game, called Project X-Ray, in which a player sees robots burst from a wall, and can shoot them using a handheld accessory.
The HoloLens videos show virtual objects generated on the fly and added to the video footage. So they are not a completely faithful representation of what a person sees through the headset. The field of view in a HoloLens headset, for instance, is actually relatively narrow, meaning virtual objects only appear directly in front of the wearer.
Speaking of Lanier’s project, Jeremy Bailenson, founding director of Stanford University’s Virtual Human Interaction Lab, says the key problem with creating virtual experiences for more than one person is tracking movement. “The critical aspect for multi-person virtual reality is that you have to track everybody’s movements very accurately in their own scene,” he says.
Bailenson believes that some form of virtual reality will prove especially useful for communications. And he says Microsoft is doing very good work with both HoloLens and its research. “Jaron is the guy to do this,” he adds. “He’s been thinking about this for decades.”