Researchers have developed a robot that uses radio waves, which can penetrate walls, to perceive hidden objects. The robot, called RF-Grasp, combines this powerful detection with more traditional computer vision to locate and grasp objects that would otherwise remain hidden. This advancement could one day streamline e-commerce fulfillment in warehouses or help a machine retrieve a screwdriver from a cluttered toolbox.

The research will be presented in May at the IEEE International Conference on Robotics and Automation. The lead author of the paper is Tara Boroushaki, a research assistant in the Signal Kinetics Group at the MIT Media Lab. Her co-authors at MIT are Adib, director of the Signal Kinetics Group, and Alberto Rodríguez, associate professor (class of 1957) in the Department of Mechanical Engineering. Other co-authors include Junshan Leng, a research engineer at Harvard University, and Ian Clester, a doctoral student at Georgia Tech.

While e-commerce continues to grow, warehouse work remains largely the domain of humans, not robots, despite sometimes hazardous working conditions. This is partly because robots struggle to locate and grasp objects in such a crowded environment. "Perception and picking are two of the hurdles the industry faces today," says Rodríguez. Using only optical vision, robots cannot perceive the presence of an item stored in a box or hidden behind another object on a shelf: visible light waves, of course, do not penetrate walls. But radio waves do.

For decades, radio-frequency identification (RFID) has been used to track all sorts of objects, from library books to pets. RFID systems have two main components: a reader and a tag. The tag is a tiny computer chip that is attached to—or, in the case of pets, implanted in—the object to be tracked. The reader emits a radio-frequency signal that is modulated by the tag and reflected back to the reader.

The reflected signal provides information about the location and identity of the tagged item. The technology has gained popularity in retail supply chains: Japan aims to use radio-frequency tracking for almost all retail purchases within a few years. Researchers realized that this proliferation of RF could be a boon for robots, as it would provide them with another mode of perception.

"Radiofrequency is a different detection method than vision," says Rodríguez. "It would be a mistake not to explore what RF can do.".

RF Grasp uses both a camera and an RF reader to find and pick up tagged objects, even when they are completely blocked from the camera's view. It consists of a robotic arm attached to a grasping hand. The camera is located at the robot's wrist. The radio frequency reader is separate from the robot and transmits tracking information to the robot's control algorithm. In this way, the robot constantly collects radio frequency tracking data and a visual image of its surroundings. Integrating these two data streams into the robot's decision-making process was one of the biggest challenges the researchers faced.

"The robot has to decide, at every moment, which of these flows is most important to think about," says Boroushaki. "It's not just about eye-hand coordination, but RF-eye-hand coordination. So the problem becomes much more complex.".

The robot initiates the search and capture process by triggering the target object's radio frequency tag to pinpoint its location. "It starts by using RF to focus its vision," Adib says. "Then it uses vision to perform precise maneuvers." The sequence is similar to hearing a siren behind you, then turning around to get a clearer picture of the siren's source.

With its two complementary senses, RF Grasp focuses on the target object. As it approaches and even begins to manipulate the object, vision, which provides much more precise detail than RF, takes over the robot's decision-making.

RF Grasp proved its effectiveness in a series of tests. Compared to a similar robot equipped only with a camera, RF Grasp was able to locate and grasp its object with half the total movement. Furthermore, RF Grasp demonstrated the unique ability to "clear up" its environment, removing packaging materials and other obstacles in its path to access the target. Rodriguez states that this demonstrates RF Grasp's "unfair advantage" over robots without penetrating radio frequency sensors. "It has this orientation that other systems simply don't have.".

RF Grasp could one day perform logistics tasks in overcrowded e-commerce warehouses. Its RF detection could even instantly verify an item's identity without handling it, exposing its barcode and then scanning it. "Radio frequency has the potential to improve some of the industry's limitations, especially regarding perception and localization," says Rodriguez.

Adib also envisions potential household applications for the robot, such as locating the correct Allen key to assemble his Ikea chair. "Or you could imagine the robot finding lost objects. It's like a super-Roomba that goes and retrieves my keys, wherever I've put them.".