Intelligent Machines

Why NASA Wants Microsoft’s HoloLens in Space

The rocket carrying two HoloLens headsets to the International Space Station blew up in June, and NASA is planning another launch.

Augmented-reality technology could be useful to astronauts in space for overlaying digital instructions atop equipment or letting them work closely with remote coworkers.

The explosion in June of the SpaceX rocket that was headed to the International Space Station felt like “a punch in the gut” to Jeff Norris, the project manager for two HoloLens projects that NASA is working on at its Jet Propulsion Laboratory (JPL) in Pasadena, California. Among the items on board were two HoloLens headsets—Microsoft’s forthcoming augmented-reality gadgets.

Astronaut Luca Parmitano uses HoloLens
Astronaut Luca Parmitano uses HoloLens at the underwater Aquarius Reef Base, which is located below the Florida Keys.

But within a couple of weeks, Norris says, his team at NASA and his counterparts at Microsoft had new HoloLens hardware that they were certifying for launch into space. That’s now scheduled to happen December 3 as part of a commercial cargo launch by the aerospace company Orbital Sciences to resupply the space station.

Here on earth, augmented-reality devices may eventually be used for a range of things like playing games that mix digital 3-D creatures with reality or talking with remote friends as if they’re in your living room. But NASA sees a number of practical—and possibly time-saving—uses for the technology in space.

NASA hopes to use HoloLens aboard the space station to allow astronauts to work with a remote expert who can see what the astronaut sees and help with unfamiliar tasks. The device might also act as an augmented-reality instruction manual that, say, uses 3-D images to show an astronaut where to place some equipment or what handle to turn. (Microsoft CEO Satya Nadella recently said in an interview that HoloLens will be available to developers within the next year; the timing of a consumer release is still unknown.)

In late July and early August, NASA astronauts tried using HoloLens to help with several tasks at the Aquarius Reef Base.

Norris, who is also the leader of the Ops Lab at JPL, says NASA is also working on other applications for HoloLens, like using augmented reality for inventory management. Apparently keeping track of where things are and how to find them is a big challenge on the space station, even though objects have bar codes on them and are organized with a database. NASA has prototyped an app that can be used to recognize an object and show the HoloLens wearer a path to follow that leads to where the object should be stored, Norris says.

In the meantime, to get some sense of what it will be like to use HoloLens on the space station, NASA experimented with HoloLens at the Aquarius underwater research station off the coast of Key Largo, Florida, in late July and early August. Astronauts used the device for tasks like checking emergency breathing equipment by going through a series of steps ranging from turning valves to finding and plugging in equipment, and setting up equipment to support an undersea robot.

In both cases, an expert sitting in a remote control center on dry land helped by using a Skype program Microsoft built for HoloLens (see “Reality Check: Comparing HoloLens and Magic Leap”) in which a forward-facing camera on the HoloLens let the expert see what the astronaut saw. If needed, the remote expert could draw in midair to point out things that the astronauts would see with the HoloLens headset (the whole time, the astronauts could also see a floating video of the expert in front of their faces). Norris thinks the task would have taken “many times as long” if it had simply been spelled out as a procedure to follow.

Though he thinks it can be helpful, Norris also says there are “enormous challenges” associated with building augmented-reality applications, such as figuring out how an application menu should look and how the user should interact with it when it’s not shown on a laptop or smartphone screen.

“The rules are different when you’re now rendering information all around a person,” he says.

The latest Insider Conversation is live! Listen to the story behind the story.

Subscribe today
Already a Premium subscriber? Log in.

Uh oh–you've read all of your free articles for this month.

Insider Premium
$179.95/yr US PRICE

More from Intelligent Machines

Artificial intelligence and robots are transforming how we work and live.

Want more award-winning journalism? Subscribe to Insider Premium.
  • Insider Premium {! insider.prices.premium !}*

    {! insider.display.menuOptionsLabel !}

    Our award winning magazine, unlimited access to our story archive, special discounts to MIT Technology Review Events, and exclusive content.

    See details+

    What's Included

    Bimonthly home delivery and unlimited 24/7 access to MIT Technology Review’s website.

    The Download. Our daily newsletter of what's important in technology and innovation.

    Access to the Magazine archive. Over 24,000 articles going back to 1899 at your fingertips.

    Special Discounts to select partner offerings

    Discount to MIT Technology Review events

    Ad-free web experience

    First Look. Exclusive early access to stories.

    Insider Conversations. Listen in as our editors talk to innovators from around the world.

/
You've read all of your free articles this month. This is your last free article this month. You've read of free articles this month. or  for unlimited online access.