Skip to Content

A robot has figured out how to use tools

In a startling demonstration, the machine drew on experimentation, data, and observation of humans to learn how simple implements could help it achieve a task.
April 11, 2019

Learning to use tools played a crucial role in the evolution of human intelligence. It may yet prove vital to the emergence of smarter, more capable robots, too.

New research shows that robots can figure out at least the rudiments of tool use, through a combination of experimenting and observing people.

Chelsea Finn, a researcher at Google Brain, and Sergey Levine, an assistant professor at UC Berkeley, developed the robotic system together with several of Levine’s students. (Finn and Levine were named Innovators Under 35 by MIT Technology Review in 2018 and 2016, respectively.)

The setup consists of an off-the-shelf robot arm that can be controlled by a person or a computer. It also includes a camera that sees the environment within reach of the arm—and, most important, a computer running a very large neural network that lets the robot learn. 

The robot worked out how to make use of simple implements, including a dustpan and broom and a duster, to move other objects around. The work hints at how robots might someday learn to perform sophisticated manipulations, and solve abstract problems, for themselves. “It’s exciting because it means the robot can figure out what to do with a tool in situations it hasn’t seen before,” Finn says. “We really want to study that sort of generality, rather than a robot learning to use one tool.”

The researchers have previously shown how a robot can learn to move objects without explicit instruction. By observing and experimenting, the robot develops a simple model of cause and effect (“Push an object this way, and it’ll end up over there”). The new robot learns in a similar way, but it builds a more complex model of the physical world (“Moving this item can move those other items over there”).

The robotic system learns in several ways. To get a basic understanding of cause and effect, it experiments with objects on its own, nudging them around to see the results. It is also fed data from lots of previous robot learning. Throughout, a recurrent neural network learns to predict what will happen in a scene if the robot takes a particular action.

To master tool use, the robot also observes human behavior. Combining its lessons from the two types of learning then lets the robot determine how to use an object in a new situation.

Annie Xie, an undergraduate student at UC Berkeley involved with the project, writes about the work in a related blog post: “With a mix of demonstration data and unsupervised experience, a robot can use novel objects as tools and even improvise tools in the absence of traditional ones.”

Levine, a leading researcher in robotic learning, says he was surprised by the robot’s ability to improvise. In one case, for example, the robot decided that a water bottle, because of its shape and size, could be used to sweep objects across a surface. 

“When you show it things that aren’t actually tools, it could come up with ways to use them that were a little bit surprising,” Levine says.

Keep Reading

Most Popular

This new data poisoning tool lets artists fight back against generative AI

The tool, called Nightshade, messes up training data in ways that could cause serious damage to image-generating AI models. 

Rogue superintelligence and merging with machines: Inside the mind of OpenAI’s chief scientist

An exclusive conversation with Ilya Sutskever on his fears for the future of AI and why they’ve made him change the focus of his life’s work.

Data analytics reveal real business value

Sophisticated analytics tools mine insights from data, optimizing operational processes across the enterprise.

How to fix the internet

If we want online discourse to improve, we need to move beyond the big platforms.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.