MIT researchers have developed a system that could bring deep learning neural networks to new—and much smaller—places, such as the tiny computer chips in wearable medical devices, household appliances, and the other 250 billion objects that make up the "Internet of Things" (IoT).

The system, called MCUNet, designs compact neural networks that offer unprecedented speed and accuracy for deep learning in IoT devices, despite limited memory and processing power. The technology could facilitate the expansion of the IoT ecosystem while saving energy and improving data security.

The research will be presented at next month's Neural Information Processing Systems Conference. The lead author is Ji Lin, a doctoral student in Song Han's lab in MIT's Department of Electrical Engineering and Computer Science. Co-authors include Han and Yujun Lin of MIT, Wei-Ming Chen of MIT and National Taiwan University, and John Cohn and Chuang Gan of the MIT-IBM Watson Artificial Intelligence Lab.

The Internet of Things

The IoT was born in the early 1980s. Graduate students at Carnegie Mellon University, including Mike Kazar in '78, connected a Coca-Cola machine to the internet. The group's motivation was simple: laziness. They wanted to use their computers to confirm the machine was stocked before leaving their office to make a purchase. It was the world's first internet-connected device. "This was more or less treated as the punchline to a joke," says Kazar, now a Microsoft engineer. "Nobody expected billions of devices on the internet."

From that Coca-Cola machine, everyday objects are becoming increasingly connected in the burgeoning IoT. That includes everything from wearable heart monitors to smart refrigerators that tell you when you're low on milk. IoT devices often run on microcontrollers—simple computer chips without an operating system, with minimal processing power, and less than one-thousandth the memory of a typical smartphone. So pattern recognition tasks like deep learning are difficult to run locally on IoT devices. For complex analysis, IoT-collected data is often sent to the cloud, making it vulnerable to hacking.

"How do we deploy neural networks directly into these small devices? It's a new area of ​​research that's gaining a lot of traction," says Han. "Companies like Google and ARM are working in this direction.".

With MCUNet, Han's group coded two components necessary for "tiny deep learning"—the operation of neural networks on microcontrollers. One of the components is TinyEngine, an inference engine that manages resources, similar to an operating system. TinyEngine is optimized to run a particular neural network architecture, which is selected by the other component of MCUNet: TinyNAS, a neural architecture search algorithm.

System algorithm coding

Designing a deep network for microcontrollers is not easy. Existing neural architecture search techniques start with a large set of possible network structures based on a predefined template, and then gradually find the one with high accuracy and low cost. "It can work quite well for GPUs or smartphones," says Lin. "But it has been difficult to directly apply these techniques to tiny microcontrollers because they are too small."

So Lin developed TinyNAS, a neural architecture search method that creates custom-sized networks. "We have many microcontrollers that come with different power capabilities and different memory sizes," says Lin. "So we developed the [TinyNAS] algorithm to optimize the search space for different microcontrollers." The custom nature of TinyNAS means it can generate compact neural networks with the best possible performance for a given microcontroller, without unnecessary parameters. "Then we deliver the final, efficient model to the microcontroller," says Lin.

To run that tiny neural network, a microcontroller also needs a poor inference engine. A typical inference engine has a certain amount of dead weight—instructions for tasks it can rarely execute. The extra code isn't a problem for a laptop or smartphone, but it could easily overwhelm a microcontroller. "It has no off-chip memory, and it doesn't have a disk," says Han. "All together it's just one megabyte of flash, so we have to handle such a small resource very carefully." Enter TinyEngine.

The researchers developed their inference engine in conjunction with TinyNAS. TinyEngine generates the essential code needed to run TinyNAS's custom neural network. Any unnecessary code is discarded, reducing compilation time. "We keep only what we need," says Han. "And because we designed the neural network, we know exactly what we need. That's the advantage of system algorithm code." In tests by the TinyEngine group, the size of the compiled binary code was between 1.9 and five times smaller than comparable microcontroller inference engines from Google and ARM. TinyEngine also contains runtime-reducing innovations, including in-place depth convolution, which cuts peak memory usage by nearly half. After coding TinyNAS and TinyEngine, Han's team tested MCUNet.

MCUNet's first challenge was image classification. Researchers used the ImageNet database to train the system with labeled images, and then to test its ability to classify new ones. On a commercial microcontroller they tested, MCUNet successfully classified 70.7 percent of the new images—the previous state-of-the-art neural network and inference engine combination had only 54 percent accuracy. "Even a 1 percent improvement is considered significant," says Lin. "So this is a big leap for microcontroller tuning."

The team found similar results in ImageNet tests of three other microcontrollers. And in both speed and accuracy, MCUNet outperformed the competition in auditory and visual "wake-up" tasks, in which a user initiates interaction with a computer using voice cues (think: "Hey Siri") or simply by entering a room. The experiments highlight MCUNet's adaptability to numerous applications.

"Enormous potential"
The promising test results give Han hope that it will become the new industry standard for microcontrollers. "It has enormous potential," he says.

The advance "pushes the frontier of deep neural network design even further into the computational domain of small, energy-efficient microcontrollers," says Kurt Keutzer, a computer scientist at the University of California, Berkeley, who was not involved in the work. He adds that MCUNet could "bring intelligent computer vision capabilities to even the simplest kitchen appliances, or enable smarter motion sensors.".

MCUNet could also make IoT devices more secure. "A key advantage is preserving privacy," says Han. "You don't need to transmit the data to the cloud.".

Analyzing data locally reduces the risk of personal information being stolen, including personal health data. Han envisions smartwatches with MCUNet that not only detect users' heart rate, blood pressure, and oxygen levels, but also analyze and help understand that information. MCUNet could also bring deep learning capabilities to IoT devices in vehicles and rural areas with limited internet access.

Furthermore, MCUNet's small computing footprint translates into a small carbon footprint. "Our big dream is a green IoT," says Han, adding that training a large neural network can burn the carbon equivalent of the lifetime emissions of five cars. MCUNet on a microcontroller would require a tiny fraction of that energy. "Our ultimate goal is to enable tiny, efficient AI with less computing power, fewer people, and less data," says Han.

Author: Daniel Ackerman, MIT