Skip to Content

Robot, Get the Fork Out of My Sink

Better vision and manipulation could help robots move into factories and even homes.
October 18, 2016

The robots are coming—to do the dishes, if we’re lucky.

Speaking at EmTech MIT 2016 today, Stefanie Tellex, an assistant professor at Brown University, showed how her research group taught a robot to perceive and grasp difficult objects, including a fork in a sink of running water, using a clever camera trick and a powerful machine-learning process.

The simple task of grabbing a fork from a sink filled with running water is an important step forward for robot-kind. Robots still struggle to perform many tasks reliably, such as grasping unfamiliar objects, especially when lighting conditions are challenging. “Most robots can’t handle most objects most of the time,” says Tellex. “That’s the hard problem we’re solving.”

The project also points to ways that robots can learn how to take on new jobs in industry and around the home. There is a huge opportunity for robots to help with elder care, for example, if they can be programmed to perform reliably in messy and ever-changing domestic situations.

Tellex’s team used an off-the-shelf industrial robot from Rethink Robotics, which has a camera in its arm, to perform the fork-grasping trick. By moving the camera and combining different images, they were able to build a virtual light-field camera, meaning it captures not just the intensity of light but also the direction of individual rays. This made it possible to build a 3-D model of the scene, and to cope with problems like reflectivity.

In separate experiments, the group is also using a machine-learning approach known as reinforcement learning to train robots to pick up unfamiliar objects. This involves letting a robot that is controlled by a large neural network experiment with different grasps and reinforcing behavior that seems to produce positive results. The results can be impressive, making it possible for a machine to devise a strategy for grasping previously unseen objects that would be extremely difficult to program manually. Tellex is also exploring ways for robots to share what they have learned, something that promises to accelerate the training process dramatically (see “10 Breakthrough Technologies: Robots That Teach Each Other”).

“Our approach is to make the robot learn to adapt itself to the environment it finds itself in,” says Tellex. “Through that learning it can reach a reliability that wasn’t possible previously. It can also use this information to generalize to other situations.”

New robot learning approaches are rapidly moving into industrial settings. Existing robotics companies are developing products that will use reinforcement learning to accelerate robot programming (see “A Japanese Robot Giant Gives Its Arms Some Brains”). Companies specializing in AI and machine learning also see the technology as a way to break into an industry that looks set to evolve rapidly (see “Google Builds a Robotic Hive-Mind Kindergarten”).

“We want robots to be able go into factories, household environments, and manipulate,” Tellex says, “and everything starts with picking something up.”

Keep Reading

Most Popular

DeepMind’s cofounder: Generative AI is just a phase. What’s next is interactive AI.

“This is a profound moment in the history of technology,” says Mustafa Suleyman.

What to know about this autumn’s covid vaccines

New variants will pose a challenge, but early signs suggest the shots will still boost antibody responses.

Human-plus-AI solutions mitigate security threats

With the right human oversight, emerging technologies like artificial intelligence can help keep business and customer data secure

Next slide, please: A brief history of the corporate presentation

From million-dollar slide shows to Steve Jobs’s introduction of the iPhone, a bit of show business never hurt plain old business.

Stay connected

Illustration by Rose Wong

Get the latest updates from
MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Thank you for submitting your email!

Explore more newsletters

It looks like something went wrong.

We’re having trouble saving your preferences. Try refreshing this page and updating them one more time. If you continue to get this message, reach out to us at with a list of newsletters you’d like to receive.