Skip to Content

Microsoft Researchers Are Working on Multi-Person Virtual Reality

Researchers in Jaron Lanier’s lab at Microsoft are exploring ways for people to share the experience of mixed reality.
October 12, 2015
In an example of shared mixed reality, two students see a visual effect as they shake hands.

Augmented or mixed reality, which renders virtual images in a view of the real world, can be spectacular to experience. But it may be even more fun when you bring a friend.

Researchers in the Microsoft lab of Jaron Lanier are experimenting with multi-person augmented reality, and the results of their work could help shape the way the technology is commercialized and used. Lanier was a pioneer of virtual and augmented, or mixed, reality in the 1980s.

Microsoft is testing a commercial augmented reality product, called HoloLens. Lanier stresses that his work is separate from HoloLens and does not reflect how that product will develop. Still, multi-person mixed reality is a long-standing challenge for those interested in the technology. Beyond gaming, there is hope that virtual and augmented reality could prove useful for communications, collaboration, and for new ways of accessing and handling information.

One of Lanier’s students, Andrea Stevenson Won, wears a headset developed in his lab at Microsoft.

Lanier’s project is called Comradre (and pronounced “comradery”). A video produced by Lanier’s lab shows several projects developed by student interns in which more than one person interacts with the same virtual object or phenomena. These headsets were made using smartphones and laptop computers, and the tracking of a person’s head movements are performed using external sensors. And most of the footage was shot through one of the headsets.

The video shows a number of experimental applications developed for the platform. Andrea Stevenson Won of Stanford University is testing the social implications of the technology with a system that highlights physical interactions with virtual effects. Judith Amores and Xavie Benavides from the MIT Media Lab created a way for real-world blocks to correspond with virtual objects and animations, offering a way for children to play in mixed reality, even if not everyone is wearing a headset.

Other students in the group created more utilitarian uses. Andrzej Banburski of the Perimeter Institute for Advanced Physics, for example, developed a tool for visualizing mathematical equations. And Kishore Rathinavel, from the University of North Carolina, Chapel Hill, created a system that allows sound waves to be detected and then shown visually. In one scene from the group’s video, the sound produced by Lanier’s flute playing is shown in mixed reality. Two other students, Victor Mateevitsi, at the University of Illinois at Chicago, and Gheric Speiginer of Georgia Tech, were also part of the group.

Jaron Lanier

Both augmented reality and virtual reality—which provides a completely immersive experience—are currently hot areas of research and commercial development at several major technology companies. Facebook, Google, and Sony are all developing headsets and software for virtual reality. Meanwhile, a company called Magic Leap has raised more than $500 million in funding to develop a convincing form of augmented reality using holographic technology (see “Breakthrough Technologies 2015: Magic Leap”).

Microsoft demonstrated HoloLens last week at an event held for the launch of Windows 10, the latest version of its computer operating system. The demo showed an experimental game, called Project X-Ray, in which a player sees robots burst from a wall, and can shoot them using a handheld accessory.

The HoloLens videos show virtual objects generated on the fly and added to the video footage. So they are not a completely faithful representation of what a person sees through the headset. The field of view in a HoloLens headset, for instance, is actually relatively narrow, meaning virtual objects only appear directly in front of the wearer.

Speaking of Lanier’s project, Jeremy Bailenson, founding director of Stanford University’s Virtual Human Interaction Lab, says the key problem with creating virtual experiences for more than one person is tracking movement. “The critical aspect for multi-person virtual reality is that you have to track everybody’s movements very accurately in their own scene,” he says.

Bailenson believes that some form of virtual reality will prove especially useful for communications. And he says Microsoft is doing very good work with both HoloLens and its research. “Jaron is the guy to do this,” he adds. “He’s been thinking about this for decades.”

Keep Reading

Most Popular

It’s time to retire the term “user”

The proliferation of AI means we need a new word.

The problem with plug-in hybrids? Their drivers.

Plug-in hybrids are often sold as a transition to EVs, but new data from Europe shows we’re still underestimating the emissions they produce.

Sam Altman says helpful agents are poised to become AI’s killer function

Open AI’s CEO says we won’t need new hardware or lots more training data to get there.

An AI startup made a hyperrealistic deepfake of me that’s so good it’s scary

Synthesia's new technology is impressive but raises big questions about a world where we increasingly can’t tell what’s real.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.