But the recognition of natural sounds, such as cheering crowds or crashing waves, has lagged behind. This is because most automated recognition systems, whether processing audio or visual information, are the result of machine learning, in which computers search for patterns in vast collections of training data. Typically, the training data must first be manually annotated, which is prohibitively expensive for all but the most demanding applications.
However, sound recognition may be making a comeback, thanks to researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL). At the recent Neural Information Processing Systems conference, they presented a sound recognition system that outperforms its predecessors but doesn't require handwritten data during training.
Instead, the researchers trained the system on video. First, existing computer vision systems that recognize scenes and objects classify the video images. The new system found correlations between those visual categories and natural sounds.
"Computer vision has been so successful that we can transfer it to other domains," says Carl Vondrick, an MIT graduate student in electrical and computer engineering and one of the paper's first two authors. "We're capitalizing on the natural synchronization between vision and sound. We augmented it with tons of unlabeled video to learn how to understand sound.".
The researchers tested their system on two standard databases of annotated sound recordings, and it was between 13 and 15 percent more accurate than the previous best-performing system. On a dataset with 10 different sound categories, it could classify the sounds with 92 percent accuracy, and on a dataset with 50 categories, it achieved 74 percent accuracy. On those same datasets, humans achieved 96 percent and 81 percent accuracy, respectively.
"Even human beings are ambiguous," says Yusuf Aytar, the other author of the paper and a postdoc in the MIT lab of electrical and computer engineering professor Antonio Torralba. Torralba is the final co-author of the article.
"We did an experiment with Carl," Aytar says. "Carl was staring at the computer monitor and couldn't see it. He would play a recording, and I would try to guess what it was. It turns out this is very, very difficult. I could say 'interior' 'exterior,' basic guesses, but when it comes to the details—'Is it a restaurant?'—that level of detail disappears. Even for annotation purposes, the task is really difficult.".
Complementary Applications:
Because collecting and processing audio data requires far less power than collecting and processing visual data, researchers envision a sound recognition system that could be used to enhance the context sensitivity of mobile devices.
When combined with GPS data, for example, a sound recognition system could determine that a mobile phone user is in a movie theater and that the film has started, and the phone could automatically route calls to a pre-recorded outgoing message. Similarly, sound recognition could improve the situational awareness of autonomous robots.
“For example, think of a self-driving car,” Aytar says. “There’s an ambulance, and the car doesn’t see it. If it hears it, it can make future predictions about the ambulance, which route it will take, based purely on the sound.”
Visual Language
The researchers' machine learning system is a neural network, so named because its architecture resembles that of the human brain. A neural network consists of processing nodes that, like individual neurons, can only perform rudimentary calculations but are densely interconnected. Information, for example, the pixel values of a digital image, is fed into the bottom layer of nodes, which processes it and passes it to the next layer, which processes it and passes it to the next layer, and so on. The training process continuously modifies the settings of the individual nodes until the output of the final layer reliably performs some sort of classification of the data—say, identifying the objects in the image.
Vondrick, Aytar, and Torralba first trained a neural network on two large sets of annotated images: one, the ImageNet dataset, contains labeled image samples of 1,000 different objects; The other, the Places dataset created by Torralba's group, contains labeled images of 401 different types of scenes, such as a playground, a bedroom, or a lecture hall.
Once the network was trained, the researchers transferred 26 terabytes of video data downloaded from the photo-sharing site Flickr. "That's about 2 million unique videos," Vondrick says. "If you had to watch them all, it would take about two years." They then trained a second neural network on the audio from the same videos. The goal of the second network was to correctly predict the labels of objects and scenes produced by the first network.
The result was a network that could interpret natural sounds in terms of image categories. For example, it could determine that the sound of birdsong tends to be associated with forest scenes and photographs of trees, birds, birdhouses, and bird feeders.
Benchmarking.
However, to compare the sound recognition network's performance with that of its predecessors, the researchers needed a way to translate its picture language into the familiar language of sound names. So they built a simple machine learning system to associate the sound recognition network's outputs with a set of standard sound labels.
To do this, the researchers used an annotated audio database, one with 50 sound categories and about 2,000 examples. Those annotations had been supplied by humans. But it's much easier to label 2,000 examples than to label 2 million. And the MIT researchers' network, trained first on unlabeled video, significantly outperformed all previous networks trained solely on the 2,000 labeled examples.
###
Written by Larry Hardesty, MIT News Office
