There are some sights and noises that people just can’t help but notice. Indeed, research in neuroscience now suggests that the recognition of salient objects is a key part of how we make sense of our environment. But building robots that can intelligently pick out items of interest using sight or sound remains a daunting challenge. So a handful of engineers are working on a new approach called selective-attention modeling, which attempts to program robots to evaluate scenes critically as some neuroscientists believe people do.
“General scene understanding is the Holy Grail for computer vision,” says University of Southern California computer scientist Laurent Itti. Neuroscience-based algorithms, he contends, “should be the new approach.”
Several research groups, including ones at Caltech and Itti’s lab at the USC, are giving robots pan-and-tilt cameras for eyes and the ability to pick out unusual objects. The robots are designed to notice, for example, a bright purple tree house at the side of a wooded road and to do it for the same reason people would-because it stands out against a backdrop of trees. Responding to eye-catching sights enables robots to act independently when they encounter something unexpected. A robot on Mars, for example, might notice an area of the ground that is discolored and take a sample of it. In contrast, robots using more traditional vision methods would detect the discolored ground only if they had been told specifically to look for it.
Itti’s robots construct maps on the basis of such local contrasts in features as color, edges, orientation, light intensity and motion. The tree house would stand out on the orientation map because of its horizontal profile amidst the vertical trees. It would also be conspicuous on the color map. The robot would overlay these maps to build a composite that highlights any striking areas.
This technique works basically the way your brain does when you scan a “what’s wrong with this picture” type of puzzle. But robotic vision needs to do more than just pick out anomalous sights. So Itti and others, such as Caltech’s Christof Koch, are also programming robots to use the approach to find specific objects. A robot looking for that purple tree house, say, knows to put greater weight on purple objects in its color map. And when forming the composite map, it will give greater weight to the color map than to other maps; the motion map, for instance, would be irrelevant, since both the tree house and its background are mainly stationary.
The strategy differs from conventional approaches to robotic vision. One common method, for example, is called object segmentation. To learn to find a coffee cup, a robot first examines a picture of the cup. It extracts-or segments-the cup’s image out of the picture. Next, it erases anything that occludes the cup, say, a sugar bowl, and cleans up the image, maybe creating a black silhouette of the cup on a white background. The robot then estimates what the cup looks like at different scales and rotations. Only then can it conduct its search-homing in on objects that have shapes similar to the cup’s, until it finds a match.
But using object segmentation can be cumbersome, and it’s not especially reliable, says Polly Pook, vice president for research at iRobot, a robotics developer in Somerville, MA. She is impressed with the selective-attention models Itti and others are building. “I think it’s a great approach,” says Pook. “It’s simplifying the process.”
Visual selective-attention models, however, still have many shortcomings. Itti’s robots cannot tell the difference between a black laptop and a neatly folded black turtleneck, for example, since their software has no way of knowing the difference between metal and fabric. But engineers hope texture maps will one day advance visual selective-attention processing yet another step, so robots can make such distinctions.
Researchers are also thinking about incorporating data from other senses, such as touch, into the final map, although they’re not yet sure how this can be done. Any robot that interacts with its environment will benefit from having tactile senses, says Ernst Niebur, a neuroscientist at the Johns Hopkins University. Niebur feels confident that he can build a tactile map that would then feed into the final composite map. “We know the brain is capable of doing it,” he says.
The next leap is putting the robots in the field. Finding a purple tree house is small stuff compared with navigating a crowded street. That takes a robot that can quickly process and respond to multiple stimuli. But just knowing where to look and what to look out for is at least a small robotic step forward.
The inside story of how ChatGPT was built from the people who made it
Exclusive conversations that take us behind the scenes of a cultural phenomenon.
How Rust went from a side project to the world’s most-loved programming language
For decades, coders wrote critical systems in C and C++. Now they turn to Rust.
Design thinking was supposed to fix the world. Where did it go wrong?
An approach that promised to democratize design may have done the opposite.
Sam Altman invested $180 million into a company trying to delay death
Can anti-aging breakthroughs add 10 healthy years to the human life span? The CEO of OpenAI is paying to find out.
Get the latest updates from
MIT Technology Review
Discover special offers, top stories, upcoming events, and more.