Skip to Content

The Giant Who Walks Amongst Us

Far from inhibiting the imagination, Augmented Reality is allowing researchers to create story characters that can interact with readers.

Giant Jimmy Jones is a friendly, helpful giant. In fact, this book character is so helpful, he can make the sun shine on an otherwise gray village. The giant simply walks across the page, reaches up to the cloud cover and pushes it out of the sun’s way so the villagers can catch some rays.

Those light rays may be virtual, but the book this scene pops out of is not.

Using augmented reality (AR), the technology behind the interactive version of Giant Jimmy Jones, New Zealand author Gavin Bishop recently collaborated with Mark Billinghurst and his colleagues at the Human Interface Technology Laboratory New Zealand (HIT Lab NZ) to turn the book into not only a storytelling device, but also a storytelling experience.

A child can flip through its pages and read it like a conventional book. But with a handheld display and computer vision tracking technology, the child can watch the story literally come to life.

“You can see animated virtual characters overlaid on the real book pages and hear the voice of Gavin Bishop reading the story,” says Billinghurst, director of the HIT Lab NZ..

While Giant Jimmy Jones currently only exists in a lab setting, there are scores of others being developed at places such as Georgia Tech University’s Augmented Environments Laboratory.

Technology is not the hindrance to turning books into interactive devices whose readers can exist within them and manipulate their stories. The most difficult roadblock stems from the limitations of physical books, most notably the reality that embedding markers that can interact with VR-headgear is expensive and produces ugly visual results on the page. 

Virtual Stories

Unlike virtual reality that exists only within the confines of a computer-generated world, augmented reality includes virtual space digitally, seamlessly overlaid onto a real environment, explains Kelly L. Dempski, a senior researcher at Chicago-based Accenture Technology Labs.

For example, AR can be used in tandem with a child reading from a book and a computer loaded with AR software. The child, then, could virtually place characters from his story anywhere in the room.

The challenge, of course, is to get the technology to work with the user.

A wearable computer in the form of a head-mounted display (HMD) is worn like a fighter-pilot helmet, fitting over the head and eyes, and projecting images within the user’s visual field.

Improvements have led to devices that resemble glasses, but even these are unwieldy, unsanitary and limited to one user at a time. The HMDs also suffer tracking problems, or perfect registration, which means that the virtual overlay doesn’t quite match up with the physical space upon which it is projected.

That visual slip confuses the brain, making a child feel off balance, mirroring the effects of alcohol consumption, Dempski claims. The user would lose the ability to differentiate between what she’s really seeing and what’s being injected into a scene via the lens.

However, experiments with these head-mounted displays has paved the way for more practical, more commercially viable AR devices such as Personal Digital Assistants (PDA) and projectors.

A PDA was used to view Magic Book, demonstrated at SIGGRAPH 2000 by a team of researchers, including Billinghurst, then with the University of Washington’s Human Interface Technology Laboratory.

But this book’s images – of boxy buildings and a stiff, humanesque figure – weren’t very sophisticated or capable of much interaction. Instead, the 3-D visuals were like “pop-up drawings on paper,” says Ulrich Neumann, professor of computer science at the University of Southern California .

The real breakthrough for the mobile AR will come when books are loaded with “trigger symbols,” or markers, that a PDA could track and then turn into organic images leaping off the page. Doing so, though, isn’t practical and is a visual turn off to readers; the markers look like barcodes, says Ramesh Raskar, a senior research scientist with Mitsubishi’s Electric Research Labs (MERL). (Ulrich, Billinghurst and others are working on algorithms for tracking to reduce the barcode look.)

That has led researchers look at alternatives such as radio frequency identification tags (RFIDs) and touch sensors in lieu of visible computers and their components.

“The point is to hide the computer,” Raskar says. “The technology should be entirely transparent.”

Building a Better Virtual Story

Spatial augmented reality makes this possible. Instead of merely overlaying images on top of objects within a reader’s view, a projector makes objects and images appear to blend into your very world – in front, to the side and behind you.

The advantage of this technology’s use in interactive storybooks is the reader can create non-linear and event-driven stories.

Raskar says a child no longer would have to read his book page by page, and any physical action of his could change the action in and plot of the story.

“To open a book and see this animation happen is counter to anyone’s experience,” Neumann says. “This is not to replace the imagination, but to help it along a bit” in the same way film adds dimensions to stories.

But Raskar says the future of spatial AR technology in books is limited. So AR for storytelling may leap completely off the page.

Steven Feiner, computer science professor, and his colleagues with Columbia University’s Computer Graphics and User Interface Lab and in collaboration with its Graduate School of Journalism, have turned documentary films into more interactive and educational tools.

In one of the lab’s “situated documentaries” about Columbia University’s system of underground tunnels, a viewer hears narration and sees both the campus and flag-like markers that indicate where portions of the story are located, Feiner explains.

By selecting a flag, the viewer immerses herself in a narrated, 3-D surround view of one of the university’s thousands of such tunnels.

Feiner’s documentaries, though, are hypermedia stories embedded in the real world and presented using a mobile augmented reality system.

For hands- (and head-) free presentations, projectors and touch sensors are being used by Raskar and his MERL colleagues to graphically animate physical objects.

In their lab, at the University of North Carolina at Chapel Hill, they’ve created a Taj Mahal that reflects a 24-hour light cycle; in only a few moments’ time, you can see the effect light would have on the building’s onion domes and flanking towers as a simulated sun rises and then falls.

Using the same technology, a nine-year-old girl, using a white paint brush, white piece of paper and white bird house, demonstrates how she can choose a new color from a digital palette and “paint” her paper and bird house.

“Children are extremely attracted to these displays,” Raskar says in a phone interview.

And adults – the ones with money in their pockets – are extremely attracted to demonstrations of spatial AR in advertising.

Paul Dietz, senior research scientist at the Mitsubishi Electric Research Laboratories and a colleague of Raskar’s, says its MERL lab in Cambridge is host to a music store display.

At the store’s entrance is a fancy set of speakers, playing catchy hip-hop music. Interest piqued, the shopper walks closer to the setup, and the music changes genres and the volume picks up.

Now standing next to the display and looking straight at a speaker, the shopper sees the inner workings of the hi-fi equipment itself. And if he handles a component, the display changes yet again.

Raskar explains the technology behind this exhibit, demonstrated at SIGGRAPH 2004 is similar to that in a photocopier; when an office worker walks by the copy machine, it detects her presence and clicks on.

These types of technologies, though, aren’t cheap – and currently make it difficult to move these devices into widespread public use, particularly in the book publishing world.

But physical books may not be its future form factor.

It may instead be a room – or “interactive narrative playspace” – in which a child creates his own adventure, such as KidsRoom, begun years ago at the MIT Media Laboratory.

“Our reliance on a physical book provides some limitations on the type of stories that can be told,” says Billinghurst of HIT Lab NZ in an email. “Although of course it still provides traditional writers an exciting new medium to work in.”

Deep Dive


Capitalizing on machine learning with collaborative, structured enterprise tooling teams

Machine learning advances require an evolution of processes, tooling, and operations.

The race to destroy PFAS, the forever chemicals 

Scientists are showing these damaging compounds can be beat.

How scientists are being squeezed to take sides in the conflict between Israel and Palestine

Tensions over the war are flaring on social media—with real-life ramifications.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.