But a new technique developed by MIT researchers would require only a handful of human demonstrations to reprogram the robot. This machine-learning method allows the robot to pick up and place unseen objects in random positions it has never encountered before. In 10 or 15 minutes, the robot will be ready to perform a new pick-and-place task.

The technique uses a neural network specifically designed to reconstruct the shapes of 3D objects. With just a few demonstrations, the system uses what the neural network has learned about 3D geometry to grasp new objects similar to those in the demonstrations.

In simulations and using a real robotic arm, the researchers demonstrate that their system can effectively manipulate never-before-seen cups, bowls, and bottles arranged in random poses, using only 10 demonstrations to teach the robot.

"Our main contribution is the overall ability to more efficiently provide new skills to robots that need to operate in more unstructured environments where there might be a lot of variability. The concept of generalization by construction is a fascinating capability because this problem is usually much more difficult," says Anthony Simeonov, a graduate student in electrical and computer engineering (EECS) and co-author of the paper.

Simeonov co-authored the article with Yilun Du, an EECS graduate student; Andrea Tagliasacchi, a Google Brain research scientist; Joshua B. Tenenbaum, the Paul E. Newton Career Development Professor of Cognitive Science and Computation in the Department of Cognitive and Brain Sciences and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); Alberto Rodriguez, an associate professor from the class of 1957 in the Department of Mechanical Engineering; and lead authors Pulkit Agrawal, a professor at CSAIL, and Vincent Sitzmann, an incoming assistant professor at EECS. The research will be presented at the International Conference on Robotics and Automation.

Grip geometry

A robot may be trained to pick up a specific object, but if that object is lying on its side (perhaps it has fallen), the robot perceives it as a completely new scenario. This is one of the reasons why machine learning systems find it so difficult to generalize to new object orientations.

To overcome this challenge, researchers created a new type of neural network model, a neural descriptor field (NDF), which learns the 3D geometry of a class of objects. The model calculates the geometric representation of a specific item using a 3D point cloud, which is a set of data points or coordinates in three dimensions. The data points can be obtained from a depth camera that provides information about the distance between the object and a viewpoint. Although the network was trained in simulation with a large dataset of synthetic 3D shapes, it can be applied directly to real-world objects.

The team designed the NDF with a property known as equidistance. With this property, if the model is shown an image of a cup upright, and then an image of the same cup sideways, it understands that the second cup is the same object, only rotated.

"This ambiguity is what allows us to handle much more effectively the cases in which the object being observed is in an arbitrary orientation," says Simeonov.

As the NDF learns to reconstruct the shapes of similar objects, it also learns to associate related parts of those objects. For example, it learns that the handles of cups are similar, even if some cups are taller or wider than others, or have shorter or longer handles.

"If you wanted to do this with another approach, you would have to manually label all the parts. Instead, our method automatically discovers these parts from the reconstruction of the shape," says Du.

Researchers use this trained NDF model to teach a robot a new skill with just a few physical examples. They move the robot's hand over the part of an object they want it to grasp, such as the rim of a bowl or the handle of a cup, and record the location of its fingertips.

Because NDF has learned a lot about 3D geometry and how to reconstruct shapes, it can deduce the structure of a new shape, allowing the system to transfer the proofs to new objects in arbitrary poses, Du explains.

Choose a winner

They tested their model in simulations and on a real robotic arm using cups, bowls, and bottles as objects. Their method achieved an 85% success rate in tasks involving picking up and placing novel objects in new orientations, while the best baseline only managed a 45% success rate. Success is defined as grasping a new object and placing it in a specific location, such as hanging cups on a shelf.

Many baselines use 2D image information instead of 3D geometry, making it more difficult for these methods to incorporate equidistance. This is one of the reasons why the NDF technique worked much better.

Although the researchers are satisfied with its performance, their method only works for the specific category of objects it was trained on. A robot taught to pick up cups will not be able to pick up boxes or headphones, as these objects have geometric characteristics too different from those of the trained network.

"In the future, the ideal would be to expand it to many categories or to abandon the notion of category altogether," says Simeonov.

They also plan to adapt the system to non-rigid objects and, in the long term, allow the system to perform selection and placement tasks when the target area changes.

This work is funded, in part, by the Defence Advanced Research Projects Agency, the Singapore Defence Science and Technology Agency, and the National Science Foundation.

###

Written by Adam Zewe, MIT News Office