Skip to Content

Look, Listen, Walk

“Augmented reality” researchers are using location-aware handhelds to change the way we experience the world around us.

You’ve seen them. Maybe you’re one of them. They’re the zombies of the New Media Era: the unthinking, the unseeing, the undead. They are all around us.

  • The man who sits on the subway, his headphones obscuring his hearing, so closed off from the people next to him that he starts singing out loud.
  • The woman talking on her cell phone walking down the street, her eyes half shut, her thoughts miles away, until she sinks up to her ankles in a puddle of melting snow.
  • The man in the coffee shop who has his laptop screen up more to shield himself from having to engage in conversation with amiable strangers than to get any work done.
  • The person driving home from work who gets so wrapped up in the chatter of talk radio that he doesn’t notice when he drives past his exit.

These people are using mobile technology to cut themselves off from the world.

What if we could use these same technologies to engage with the world more fully? That’s a goal of research and experimentation in “augmented reality.”

People have been talking about virtual reality for decades now-the idea that we can create imaginary environments that engage all of our senses and that we can move through as if they were real environments. Augmented reality turns that premise on its head-heightening our awareness of the real world by annotating it with information conveyed by mobile technologies. Augmented reality has powerful new applications for education, tourism, and storytelling.

In early February, a powerful demonstration of augmented reality took place at Boston’s Museum of Science. Eric Klopfer, an MIT professor of urban studies and planning, along with a team of researchers from the Education Arcade (an MIT-based consortium devoted to promoting the pedagogical use of computer and video games) conducted what they called “a Hi-Tech Who Done It.” The activity was designed for middle-school kids and their parents. Participants were assigned to teams, consisting of three adult-child pairs, and given a handheld. For the next few hours, they would search high and low for clues of the whereabouts and identity of the notorious Pink Flamingo Gang. Thieves have stolen an artifact and substituted a fake in its place. Thanks to museum’s newly installed Wi-Fi network and the players’ location-aware handhelds, each gallery offered the opportunity to interview cyber-suspects, download objects, examine them with virtual equipment, and trade their findings.

Each parent-child unit was assigned a different role-biologists, detectives, or technologists-enabling them to use different tools on the evidence they gathered. As I followed the eager participants about the museum, they used walkie-talkies to share information and to call impromptu meetings to compare notes; at one point, a hyperventilating sixth grade girl lectured some other kid’s parents about what she learned about the modern synthetic material found in the sample picked up near the shattered mummy case. Racing against time and against rival teams, the kids, parents in tow, sprinted from hall to hall.

I was with one of the teams when they solved the puzzle. A young girl thrust her arms in the air and shouted, “We are the smartest people in the whole museum!” What a visceral experience of empowerment! The same girl said that everyone else in her family was smart in science but that on this occasion, she felt like a genius.

Talking to the parents afterward, one woman told the research team, “This is the longest time I’ve ever spent having a substantial conversation with my son in as long as I can remember-without any fighting.” Many of the others had in the past dragged their kids to the museum kicking and screaming. This time, however, these same kids wanted to go back and spend more time looking at exhibits they had brushed past in their investigations.

The activity had forced the kids to really pay attention to what they were looking at, to ask and answer new questions, and to process the information in new ways. These kids weren’t moving in orderly lines through the science museum; they owned that space. It wasn’t a sanctuary; it was their playground.

But there was nothing chaotic about their play. This was hard work, and it engaged every corner of their brains. Though the robbery was imaginary, the kids had to go through something akin to the real-world scientific process to solve the mystery-gathering evidence, forming hypotheses, challenging each other’s interpretations, and in the end, presenting the data to the judges to see how close they came to figuring out all of the case’s nuances.

Klopfer’s museum experiment seems calm and contained compared to geo-caching, which one website calls “the sport where you are the search engine.” A treasure chest, containing a logbook and perhaps some rewards, is buried in some obscure location. The GPS coordinates are posted on the Internet and people come from all over, scrambling through the bush or grubbing in the dirt to try to track it down. When they find the cache, they take a memento, sign the log book, and leave a gift for the next searcher. Sometimes, the chest contains another set of coordinates, sending the finder scampering off for a new location. Geo-caching has become a worldwide phenomenon.

Many of the publishers of tourist guides are already experimenting with electronic versions of their texts. But they are underestimating the ways these devices can alter where we go and what we see. Take, for example, the History Unwired project-an initiative of Michael Epstein and Cristobal Garcia, entrepreneurial graduate students in MIT’s Comparative Media Studies Program. This project is working with the tourism boards in Venice and in Alcala de Henares (Cervantes’s hometown) to develop new applications for handhelds. Venice may be a city with a rich history and a unique atmosphere, but most tourists experience it as a theme park ride-hop on a gondola and drift down the main canal. The local merchants and the city officials want use handhelds to lure tourists into other parts of the city to share the wealth and avoid the wear and tear on their fragile environment. The content of the tours is being developed at a grassroots level involving discussions with local community members.

Another initiative coming out of the MIT Comparative Media Studies department allows visitors to Paris’s Marais neighborhood to view old photographs and historic illustrations of the city. This project, called Flneurs Savants, was developed by graduate students Andrea McCarty and Rekha Murthy. While the program encourages visitors to engage with the area’s modern inhabitants, Flneurs Savants also invites people to scan the buildings and landscape for traces of a world that may no longer exist but that shaped how the neighborhood developed. You could imagine a tour that worked in the other direction, mapping onto the current city the speculations of urban planners and science fiction writers about what it might look like in 50 or 100 years.
Or imagine the ways these tools could help activists and urban planners to better understand the problems confronting a local community. The handheld could encourage people walking through the Boston Common at dusk to be more aware of the homeless people sleeping on the ground, or encourage amblers to look for signs of pollution as they cross the Charles River.

We can move through these environments at our own pace, look at what interests us, and access multiple layers of annotation. Of course, you can do that with a guidebook in your hand. But the handheld delivers a media-rich experience, enabling you to access photographs, sound files, and moving images that complement what you are seeing with your own eyes.

Some researchers believe that these devices may also become effective vehicles for storytelling. Last Halloween, MIT grad student Epstein organized Ghosts at the Granary in Boston. He worked with the Freedom Trail Foundation to produce a spooky tour of a local graveyard that combined live performers interacting with animated characters that appeared to visitors on their handhelds.

Creative artists have long been fascinated with trying to tell the multiple stories that unfold in the same community and the different ways that participants see each other. Imagine a digital artist doing in a real space what Edgar Lee Masters achieved in Spoon River Anthology, or mapping onto the houses of a small town the kinds of stories Sherwood Anderson spun in Winesburg, Ohio. As we look closely at the outside of the buildings, we struggle to figure out where fiction ends and reality begins.

Writing in 1959, Kevin Lynch, a founding figure in urban studies, proposed new ways to map our experience of the city. He wanted to create tools for explaining the life of the city-tools that enabled us to see the most common routes and best remembered landmarks, that allowed us to feel the “interconnectedness” of the urban experience, and that recorded “particular zones that for any one individual might be more intensely felt.” The location-aware handheld represents such a resource. These devices may wake us from the stupor with which we so often move through our neighborhoods. They may break tourists from the tyranny of the guidebook. They may empower us to move through the space with a purpose and invite us to look at each detail from a more informed perspective.

Keep Reading

Most Popular

What to know about this autumn’s covid vaccines

New variants will pose a challenge, but early signs suggest the shots will still boost antibody responses.

DeepMind’s cofounder: Generative AI is just a phase. What’s next is interactive AI.

“This is a profound moment in the history of technology,” says Mustafa Suleyman.

Human-plus-AI solutions mitigate security threats

With the right human oversight, emerging technologies like artificial intelligence can help keep business and customer data secure

Next slide, please: A brief history of the corporate presentation

From million-dollar slide shows to Steve Jobs’s introduction of the iPhone, a bit of show business never hurt plain old business.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.