Bauby, who had signed a contract to write a book shortly before his accident, wrote his memoir, "The Diving Bell and the Butterfly," using a dictation system in which his speech therapist recited the alphabet and he blinked when she said the correct letter. They wrote the 130-page book by blinking in unison.

Technology has come a long way since Bauby's accident. Many people with severe motor impairments caused by locked-in syndrome, cerebral palsy, amyotrophic lateral sclerosis, or other conditions can communicate using computer interfaces in which they select letters or words on an on-screen grid by flipping a single switch, often by pressing a button, exhaling a breath, or blinking.

However, these row-and-column scanning systems are very rigid and, like the technique used by Bauby's speech therapist, highlight each option one at a time, making them frustratingly slow for some users. Furthermore, they are not suitable for tasks where options cannot be arranged in a grid, such as drawing, browsing the internet, or playing games.

A more flexible system developed by MIT researchers places individual selection indicators next to each option on the computer screen. The indicators can be placed anywhere—next to anything that can be clicked with the mouse—so the user doesn't have to scroll through a grid of options to make their selections. The system, called Nomon, incorporates probabilistic reasoning to learn how users select, and then adjusts the interface to improve speed and accuracy.

Participants in a user study were able to type faster with Nomon than with a row-and-column scanning system. Users also performed better on an image selection task, demonstrating that Nomon can be used for more than just typing.

"It's great and exciting to be able to develop software that has the potential to help people. Being able to find those signals and turn them into communication the way we're used to is a really interesting problem," says lead author Tamara Broderick, associate professor in MIT's Department of Electrical Engineering and Computer Science (EECS) and a member of the Information and Decision Systems Lab and the Data, Systems and Society Institute.

Along with Broderick, the team includes Nicholas Bonaker, an EECS graduate student; Emli-Mari Nel, head of innovation and machine learning at Averly and a visiting professor at the University of the Witwatersrand (South Africa); and Keith Vertanen, an associate professor at Michigan Tech. The research is being presented at the ACM Conference on Human Factors in Computer Systems.

On the clock

In Nomon's interface, there's a small analog clock next to each user-selectable option. (A gnomon is the part of a sundial that casts a shadow.) The user looks at an option and then presses its switch when the clock's hand passes a red "noon" line. After each click, the system changes the clock phases to separate the next most likely targets. The user presses repeatedly until they select their target.

When used as a keyboard, Nomon's machine learning algorithms attempt to guess the next word based on previous words and each new letter the user selects.

Broderick developed a simplified version of Nomon several years ago, but decided to revise it to make the system easier for people with motor disabilities to use. He enlisted the help of Bonaker, then an undergraduate student, to redesign the interface.

First, they consulted non-profit organizations that work with people with motor disabilities, as well as a user with motor disabilities, to gather their opinion on the design of Nomon.

Next, they designed a user study that better represented the capabilities of people with motor disabilities. They wanted to ensure they thoroughly examined the system before investing much of the valuable time of users with motor disabilities, so they first tested it with users without a switch, Broderick explains.

Changing the switch

To obtain more representative data, Bonaker devised a webcam-based switch that was more difficult to use than simply pressing a key. Users who didn't use the switch had to tilt their body to one side of the screen and then back to the other side to register a click.

"And they have to do it at the precise moment, so it really slows them down. We did some empirical studies that showed they were much closer to the response times of people with motor disabilities," Broderick says.

They conducted a 10-session user study with 13 participants without switching and one user with an advanced form of spinal muscular dystrophy. In the first nine sessions, participants used Nomon and a row-column scanning interface for 20 minutes each to input text, and in the tenth session, they used both systems for an image selection task.

Users without switching typed 15 percent faster with Nomon, while the user with a motor disability typed even faster than users without switching. When typing unfamiliar words, users were 20 percent faster overall and made half as many errors. In the final session, they were able to complete the image selection task 36 percent faster using Nomon.

"Nomon is much more forgiving than row-column scanning. With row-column scanning, even if you make a slight mistake, you've now chosen B instead of A, and that's an error," says Broderick.

Adapting to noisy clicks

Using probabilistic reasoning, Nomon incorporates everything it knows about where the user is likely to click to make the process faster, easier, and less prone to errors. For example, if the user selects "Q," Nomon will make it as easy as possible for them to select "U.".

Nomon also learns how the user clicks. So, if the user always clicks shortly after the clock hand strikes noon, the system adapts in real time. It also adapts to noise. If the user clicks frequently, the system requires additional clicks to ensure accuracy.

This probabilistic reasoning makes Nomon powerful, but it also requires more clicks than row- and column-based scanning systems. Multiple clicks can be a difficult task for users with severe motor impairments.

Broderick hopes to reduce the number of clicks by incorporating eye tracking into Nomon, giving the system more robust information about what the user might choose next based on which part of the screen they are looking at. The researchers also want to find a better way to automatically adjust clock speeds to help users be more accurate and efficient.

They are working on a new series of studies in which they plan to partner with more users with motor disabilities.

"So far, feedback from users with motor impairments has been invaluable to us; we are very grateful to the user with a motor impairment who provided feedback on our initial interface and to the user with a motor impairment who participated in our study. We are currently expanding our study to work with a larger and more diverse group from our target population. With their help, we are already introducing new improvements to our interface and working to better understand how Nomon works," he says.

This research has been funded, in part, by the Seth Teller Memorial Fund to Advanced Technology for People with Disabilities, a Peter J. Eloranta University Summer Research Fellowship, the MIT Intelligence Quest, and the National Science Foundation.

###

Written by Adam Zewe, MIT News Office