Skip to Content

Robot, Get the Fork Out of My Sink

Better vision and manipulation could help robots move into factories and even homes.
October 18, 2016

The robots are coming—to do the dishes, if we’re lucky.

Speaking at EmTech MIT 2016 today, Stefanie Tellex, an assistant professor at Brown University, showed how her research group taught a robot to perceive and grasp difficult objects, including a fork in a sink of running water, using a clever camera trick and a powerful machine-learning process.

The simple task of grabbing a fork from a sink filled with running water is an important step forward for robot-kind. Robots still struggle to perform many tasks reliably, such as grasping unfamiliar objects, especially when lighting conditions are challenging. “Most robots can’t handle most objects most of the time,” says Tellex. “That’s the hard problem we’re solving.”

The project also points to ways that robots can learn how to take on new jobs in industry and around the home. There is a huge opportunity for robots to help with elder care, for example, if they can be programmed to perform reliably in messy and ever-changing domestic situations.

Tellex’s team used an off-the-shelf industrial robot from Rethink Robotics, which has a camera in its arm, to perform the fork-grasping trick. By moving the camera and combining different images, they were able to build a virtual light-field camera, meaning it captures not just the intensity of light but also the direction of individual rays. This made it possible to build a 3-D model of the scene, and to cope with problems like reflectivity.

In separate experiments, the group is also using a machine-learning approach known as reinforcement learning to train robots to pick up unfamiliar objects. This involves letting a robot that is controlled by a large neural network experiment with different grasps and reinforcing behavior that seems to produce positive results. The results can be impressive, making it possible for a machine to devise a strategy for grasping previously unseen objects that would be extremely difficult to program manually. Tellex is also exploring ways for robots to share what they have learned, something that promises to accelerate the training process dramatically (see “10 Breakthrough Technologies: Robots That Teach Each Other”).

“Our approach is to make the robot learn to adapt itself to the environment it finds itself in,” says Tellex. “Through that learning it can reach a reliability that wasn’t possible previously. It can also use this information to generalize to other situations.”

New robot learning approaches are rapidly moving into industrial settings. Existing robotics companies are developing products that will use reinforcement learning to accelerate robot programming (see “A Japanese Robot Giant Gives Its Arms Some Brains”). Companies specializing in AI and machine learning also see the technology as a way to break into an industry that looks set to evolve rapidly (see “Google Builds a Robotic Hive-Mind Kindergarten”).

“We want robots to be able go into factories, household environments, and manipulate,” Tellex says, “and everything starts with picking something up.”

Keep Reading

Most Popular

computation concept
computation concept

How AI is reinventing what computers are

Three key ways artificial intelligence is changing what it means to compute.

still from Embodied Intelligence video
still from Embodied Intelligence video

These weird virtual creatures evolve their bodies to solve problems

They show how intelligence and body plans are closely linked—and could unlock AI for robots.

seeing is believing concept
seeing is believing concept

Our brains exist in a state of “controlled hallucination”

Three new books lay bare the weirdness of how our brains process the world around us.

We reviewed three at-home covid tests. The results were mixed.

Over-the-counter coronavirus tests are finally available in the US. Some are more accurate and easier to use than others.

Stay connected

Illustration by Rose WongIllustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.