Skip to Content

Robot, Get the Fork Out of My Sink

Better vision and manipulation could help robots move into factories and even homes.
October 18, 2016

The robots are coming—to do the dishes, if we’re lucky.

Speaking at EmTech MIT 2016 today, Stefanie Tellex, an assistant professor at Brown University, showed how her research group taught a robot to perceive and grasp difficult objects, including a fork in a sink of running water, using a clever camera trick and a powerful machine-learning process.

The simple task of grabbing a fork from a sink filled with running water is an important step forward for robot-kind. Robots still struggle to perform many tasks reliably, such as grasping unfamiliar objects, especially when lighting conditions are challenging. “Most robots can’t handle most objects most of the time,” says Tellex. “That’s the hard problem we’re solving.”

The project also points to ways that robots can learn how to take on new jobs in industry and around the home. There is a huge opportunity for robots to help with elder care, for example, if they can be programmed to perform reliably in messy and ever-changing domestic situations.

Tellex’s team used an off-the-shelf industrial robot from Rethink Robotics, which has a camera in its arm, to perform the fork-grasping trick. By moving the camera and combining different images, they were able to build a virtual light-field camera, meaning it captures not just the intensity of light but also the direction of individual rays. This made it possible to build a 3-D model of the scene, and to cope with problems like reflectivity.

In separate experiments, the group is also using a machine-learning approach known as reinforcement learning to train robots to pick up unfamiliar objects. This involves letting a robot that is controlled by a large neural network experiment with different grasps and reinforcing behavior that seems to produce positive results. The results can be impressive, making it possible for a machine to devise a strategy for grasping previously unseen objects that would be extremely difficult to program manually. Tellex is also exploring ways for robots to share what they have learned, something that promises to accelerate the training process dramatically (see “10 Breakthrough Technologies: Robots That Teach Each Other”).

“Our approach is to make the robot learn to adapt itself to the environment it finds itself in,” says Tellex. “Through that learning it can reach a reliability that wasn’t possible previously. It can also use this information to generalize to other situations.”

New robot learning approaches are rapidly moving into industrial settings. Existing robotics companies are developing products that will use reinforcement learning to accelerate robot programming (see “A Japanese Robot Giant Gives Its Arms Some Brains”). Companies specializing in AI and machine learning also see the technology as a way to break into an industry that looks set to evolve rapidly (see “Google Builds a Robotic Hive-Mind Kindergarten”).

“We want robots to be able go into factories, household environments, and manipulate,” Tellex says, “and everything starts with picking something up.”

Keep Reading

Most Popular

A Roomba recorded a woman on the toilet. How did screenshots end up on Facebook?

Robot vacuum companies say your images are safe, but a sprawling global supply chain for data from our devices creates risk.

A startup says it’s begun releasing particles into the atmosphere, in an effort to tweak the climate

Make Sunsets is already attempting to earn revenue for geoengineering, a move likely to provoke widespread criticism.

10 Breakthrough Technologies 2023

Every year, we pick the 10 technologies that matter the most right now. We look for advances that will have a big impact on our lives and break down why they matter.

These exclusive satellite images show that Saudi Arabia’s sci-fi megacity is well underway

Weirdly, any recent work on The Line doesn’t show up on Google Maps. But we got the images anyway.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.