Skip to Content

A Sidewalk Disappearing Act

Automatically removing people from street-level imagery could help prevent privacy complaints.

In 2008, responding to privacy concerns, Google started to blur the faces of people caught by car-mounted cameras and shown in its Google Street View mapping service. Researchers in California believe they have now come up with a better solution–software that automatically removes any trace that a person was in a scene.

Now you see me: Software that removes pedestrians from Google Street View could help to alleviate privacy concerns.

The approach protects people’s privacy while also providing a cleaner street-level view, says Arturo Flores, a computer-science graduate student in the Artificial Intelligence Group at the University of California, San Diego. “Even with face blurring, it is still possible to identify a person,” Flores says. Clothing, body shape, and height, combined with a location, can be enough to recognize someone, he says.

Google’s Street View vans use nine roof-mounted cameras to take regular shots of the scene around them. These are then stitched together to produce a near-seamless panoramic view. But automatically removing people from thousands of varied images, each showing different scenes, is a challenge.

Flores’s software first has to detect any pedestrians in a scene. This is done using a standard object-recognition algorithm called implicit shape model (ISM), which was developed at the Swiss Federal Institute of Technology. “The idea is to find a rough contour of pedestrians,” says Bastian Leibe, a codeveloper of ISM who is now at RWTH Aachen University. Because there is so much variability in human appearance, the algorithm takes a probabilistic approach–looking for similarities between the shapes in images and hundreds of images of pedestrians that it has been trained to recognize.

Once a pedestrian has been identified and cut from an image, the hole left behind has to be filled in. Flores’s software does this by using photographs captured before and after the image in question by Google’s Street View vans. These images show a view of the background from slightly different angles–the algorithm can reorient the background and stitch it into the space left behind by the missing pedestrian.

Flores and his advisor, Serge Belongie, recently presented the work at the IEEE International Workshop on Mobile Vision in Chicago. Flores says some real-world images are simply too unusual for the software to process properly. Odd artifacts have been left behind in some shots, such as dogs on a leash without their owner, or pairs of shoes apparently abandoned on the sidewalk.

The system also struggles to generate a background when a pedestrian is walking in the same direction as the Google van, says Flores. “It just isn’t possible to get an unobstructed view of the background,” he says.

Andrew Blake, deputy managing director of Microsoft Research Cambridge, in the U.K., and an expert in computer vision, says most of the techniques used in Flores’s software are well-known. He adds that recent breakthroughs have made it easier to stitch together images to create a smooth-looking composite. “State-of-the-art panorama-stitching software allows the photographer to stitch together the best possible panorama from any disorderly pile of pictures of a scene,” he says.

Leibe notes that this software, or something similar, could ultimately be used to create street-level imaging for turn-by-turn navigation systems, to replace the computer-generated animations now used by such systems.

Keep Reading

Most Popular

It’s time to retire the term “user”

The proliferation of AI means we need a new word.

The problem with plug-in hybrids? Their drivers.

Plug-in hybrids are often sold as a transition to EVs, but new data from Europe shows we’re still underestimating the emissions they produce.

Sam Altman says helpful agents are poised to become AI’s killer function

Open AI’s CEO says we won’t need new hardware or lots more training data to get there.

A brief, weird history of brainwashing

L. Ron Hubbard, Operation Midnight Climax, and stochastic terrorism—the race for mind control changed America forever.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.