Skip to Content

A robot has figured out how to use tools

In a startling demonstration, the machine drew on experimentation, data, and observation of humans to learn how simple implements could help it achieve a task.
April 11, 2019
ANNIE XIEAnnie Xie

Learning to use tools played a crucial role in the evolution of human intelligence. It may yet prove vital to the emergence of smarter, more capable robots, too.

New research shows that robots can figure out at least the rudiments of tool use, through a combination of experimenting and observing people.

Chelsea Finn, a researcher at Google Brain, and Sergey Levine, an assistant professor at UC Berkeley, developed the robotic system together with several of Levine’s students. (Finn and Levine were named Innovators Under 35 by MIT Technology Review in 2018 and 2016, respectively.)

The setup consists of an off-the-shelf robot arm that can be controlled by a person or a computer. It also includes a camera that sees the environment within reach of the arm—and, most important, a computer running a very large neural network that lets the robot learn. 

The robot worked out how to make use of simple implements, including a dustpan and broom and a duster, to move other objects around. The work hints at how robots might someday learn to perform sophisticated manipulations, and solve abstract problems, for themselves. “It’s exciting because it means the robot can figure out what to do with a tool in situations it hasn’t seen before,” Finn says. “We really want to study that sort of generality, rather than a robot learning to use one tool.”

The researchers have previously shown how a robot can learn to move objects without explicit instruction. By observing and experimenting, the robot develops a simple model of cause and effect (“Push an object this way, and it’ll end up over there”). The new robot learns in a similar way, but it builds a more complex model of the physical world (“Moving this item can move those other items over there”).

The robotic system learns in several ways. To get a basic understanding of cause and effect, it experiments with objects on its own, nudging them around to see the results. It is also fed data from lots of previous robot learning. Throughout, a recurrent neural network learns to predict what will happen in a scene if the robot takes a particular action.

To master tool use, the robot also observes human behavior. Combining its lessons from the two types of learning then lets the robot determine how to use an object in a new situation.

Annie Xie, an undergraduate student at UC Berkeley involved with the project, writes about the work in a related blog post: “With a mix of demonstration data and unsupervised experience, a robot can use novel objects as tools and even improvise tools in the absence of traditional ones.”

Levine, a leading researcher in robotic learning, says he was surprised by the robot’s ability to improvise. In one case, for example, the robot decided that a water bottle, because of its shape and size, could be used to sweep objects across a surface. 

“When you show it things that aren’t actually tools, it could come up with ways to use them that were a little bit surprising,” Levine says.

Keep Reading

Most Popular

Scientists are finding signals of long covid in blood. They could lead to new treatments.

Faults in a certain part of the immune system might be at the root of some long covid cases, new research suggests.

Large language models can do jaw-dropping things. But nobody knows exactly why.

And that's a problem. Figuring it out is one of the biggest scientific puzzles of our time and a crucial step towards controlling more powerful future models.

OpenAI teases an amazing new generative video model called Sora

The firm is sharing Sora with a small group of safety testers but the rest of us will have to wait to learn more.

Google’s Gemini is now in everything. Here’s how you can try it out.

Gmail, Docs, and more will now come with Gemini baked in. But Europeans will have to wait before they can download the app.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at customer-service@technologyreview.com with a list of newsletters you’d like to receive.