Taking a step in this direction, MIT researchers developed an image dataset that allows for the simulation of peripheral vision in machine learning models. They discovered that training models with this dataset improved the models' ability to detect objects in the visual periphery, although the models still performed worse than humans.
Their results also revealed that, unlike humans, neither the size of objects nor the amount of visual clutter in a scene had a major impact on AI performance.
"Something fundamental is happening here. We've tested many different models, and even when we train them, they improve a little, but they're not exactly like humans. So the question is: what are these models missing?"says Vasha DuTell, a postdoctoral researcher and co-author of a paper detailing this study.
Answering this question could help researchers create machine learning models capable of seeing the world as humans do. In addition to improving driver safety, these models could be used to develop easier-to-read displays.
Furthermore, a deeper understanding of peripheral vision in AI models could help researchers better predict human behavior, adds lead author Anne Harrington MEng '23.
"Modeling peripheral vision, if we can truly capture the essence of what is depicted in the periphery, can help us understand the characteristics of a visual scene that cause our eyes to move to gather more information," he explains.
Co-authors include Mark Hamilton, a graduate student in Electrical and Computer Engineering; Ayush Tewari, a postdoctoral fellow; Simon Stent, research director at the Toyota Research Institute; and lead authors William T. Freeman, Thomas and Gerd Perkins Professor of Electrical and Computer Engineering and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); and Ruth Rosenholtz, a senior research scientist in the Department of Cognitive and Brain Sciences and a member of CSAIL. The research will be presented at the International Conference on Representational Learning.
"When a human interacts with a machine (a car, a robot, a user interface), it's very important to know what that person can see. Peripheral vision plays a fundamental role in that understanding," says Rosenholtz.
Simulate peripheral vision
Extend your arm forward and raise your thumb: the small area around your thumbnail is what the fovea sees—the tiny depression in the center of the retina that provides the sharpest vision. Everything else is in the visual periphery. The visual cortex renders a scene with less detail and reliability as you move away from that point of sharp focus.
Many of the existing approaches to modeling peripheral vision in AI represent this deterioration of detail by blurring the edges of the images, but the loss of information that occurs in the optic nerve and visual cortex is much more complex.
To achieve a more precise approach, the MIT researchers started with a technique used to model peripheral vision in humans. This method, known as texture mosaic modeling, transforms images to represent the loss of visual information in humans.
They modified this model so that it could transform images in a similar way, but in a more flexible way that did not require knowing in advance where the person or AI would point their eyes.
"That allowed us to faithfully model peripheral vision in the same way it is done in human vision research," Harrington says.
The researchers used this modified technique to generate a huge dataset of transformed images that appear more textured in certain areas, to represent the loss of detail that occurs when a human looks further out towards the periphery.
Next, they used the dataset to train several computer vision models and compared their performance to that of humans in an object detection task.
"We had to be very clever in setting up the experiment so we could also test it on machine learning models. We didn't want to have to retrain the models on a toy task they weren't supposed to do," he explains.
Peculiar performance
The humans and the models were shown pairs of transformed images that were identical except that one of the images had a target object located in the periphery. Each participant was then asked to choose the image with the target object.
"One thing that really surprised us was how good people were at detecting objects in the periphery. We went through at least 10 different sets of images that were too easy. We had to use increasingly smaller objects," Harrington adds.
The researchers found that training models from scratch on their dataset resulted in the greatest performance boost and improved the ability to detect and recognize objects. Refining a model on their dataset—a process that involves fine-tuning a pre-trained model to perform a new task—led to smaller performance gains.
But in all cases, the machines were not as good as humans, and they were especially bad at detecting objects in the far periphery. Their performance also did not follow the same patterns as that of humans.
"That could suggest that the models aren't using context in the same way humans do to perform these detection tasks. The models' strategy might be different," Harrington says.
The researchers plan to continue exploring these differences with the goal of finding a model that can predict human performance in peripheral vision. This could allow, for example, artificial intelligence systems to alert drivers to hazards they might otherwise miss. They also hope to inspire other researchers to conduct further computer vision studies using their publicly available dataset.
This work has been funded, in part, by the Toyota Research Institute and the MIT CSAIL METEOR Fellowship.
###
Written by Adam Zewe, MIT News
