Hyperlinking Reality via Phones

Nokia researchers are working on a system that allows physical objects to be identified and connected to the Internet through mobile-phone screens.

A Nokia research project could one day make it easier to navigate the real world by superimposing virtual information on an image of your surroundings. The new software, called Mobile Augmented Reality Applications (MARA), is designed to identify objects viewed on the screen of a camera phone.

A prototype Nokia camera phone, equipped with sensors and software called MARA, can superimpose virtual information and hyperlinks onto a real-world scene. Credit: Nokia Research Center

The Nokia research team has demonstrated a prototype phone equipped with MARA software and the appropriate hardware: a global positioning system (GPS), an accelerometer, and a compass. The souped-up phone is able to identify restaurants, hotels, and landmarks and provide Web links and basic information about these objects on the phone's screen. In addition, says David Murphy, an engineer at Nokia Research Center, in Helsinki, Finland, who works on the project, the system can also be used to find nearby friends who have phones with GPS and the appropriate software.

The field of augmented reality, in which supplementary information from a computer or the Internet is overlaid onto the real world, has been the topic of science fiction and serious academic and military study for years. Historically, augmented-reality systems have required small backpacks with computing and networking hardware that stream information onto a visual display. But in recent years, researchers have been experimenting with more consumer-friendly ways to augment reality.

Mobile phones, in particular, are an appealing gateway to the virtual world. Their computing capabilities have increased substantially, and a growing number are GPS-enabled and can access high-speed data networks.

For the MARA project, Murphy and Nokia researcher Markus Kähäri outfitted a Nokia 6680 mobile device with a box containing extra hardware: a GPS sensor to determine the location of the phone, a three-access accelerometer to determine the orientation of the phone's camera (which could be directed at a building or the ground, for instance), and a compass .

Once the phone is in camera mode and capturing a video stream, Murphy explains, MARA pulls together the information from the three sensors to pinpoint the location and orientation of the phone. The software then scours a database of objects--which can be loaded onto a phone or can be accessed through a network connection--to determine which object would be visible to the camera. Once visibility is determined, MARA highlights the objects and provides extra information and hyperlinks if available. So, if a nearby restaurant is in the database and within view, the software could display the menu and wait time, and by clicking on the hyperlink, you could visit the restaurant's website.

This capability becomes particularly compelling when people, as well as buildings, are incorporated into the database. If you have a GPS sensor in your mobile device and elect to share your location, Murphy says, people could "click on you to link to your blog." He adds, "You could go to a football match and be able to see information on the players, or ball movement, or tactics by looking at the field with your device."

MARA has an additional feature, says Murphy. To access a satellite view of your location and nearby landmarks, simply point the phone's camera at the ground. The software infers the orientation and displays the map.