Faculty Member Earns NSF CAREER Award for Computer Vision and AI Research

Self-driving vehicles are expected to be a standard in the not-too-distant future, but Georgia Tech assistant professor Judy Hoffman said the technology still has some bumps it needs to get over before that happens.

Hoffman recently received the National Science Foundation’s CAREER Award, which will fund her research in computer vision and artificial intelligence (AI) for the next five years. She earned the award through her proposal, Vision Systems for an Evolving World, a project that seeks to solve the dilemma of when a computer vision artificial intelligence struggles to adapt to new environments and circumstances.

In a self-driving vehicle, cameras are stationed on all sides of the body, while an embedded AI system scans and processes the incoming data stream to continuously inform itself of appropriate actions to take. These AI systems are taught by computer scientists to recognize things like a car switching lanes or a pedestrian walking through a crosswalk, and to slow down or stop the vehicle when necessary. Hoffman said learning models today are trained through large collections of data that are often manually collected, curated, and annotated at great expense.

These systems process millions of images to learn how to operate a vehicle, but there are always scenarios that the scientists who train them can’t anticipate. And when it comes to something that travels at a high rate of speed, the inability to process a new piece of data can create a dangerous liability.

“When you move to a new city, street signs look different, or you drive in a rural area, and the road layout looks a little different, or you drive in snowy conditions or foggy conditions,” Hoffman said. “If it hasn’t seen all those examples beforehand, the model’s ability to continue to perform well and recognize the things it’s supposed to recognize can degrade.

“The thing we’re addressing is this issue that systems today are brittle, and they are highly susceptible to that initial phase of learning. You want to make sure that they’re more flexible, that they can be more reliable in circumstances and adapt on their own if they need to.”

If the human training the computer can’t think through every possible scenario, then the only way to make self-driving vehicles a reality is to create an AI that can recognize new environments and adapt to them without human interference or assistance. Hoffman said her new project will explore methods that make the learning systems as robust as possible so they can automatically incorporate new examples of observations.

Image

“The goal of this project is to be able to develop vision systems that are resilient to the ways in which the visual appearance of the world can change,” she said.

“The technology is already there for most circumstances. It’s how do we get that end piece and do it in a reliable way. That’s the key challenge. Because those circumstances are inherently rare, you’re not going to have them in your core training data, and you need to be able to handle them as they arrive.”

To fill in the gaps, scientists can start with objects that frequently show up in images, which Hoffman defines as landmarks. When enough landmarks are identified, the AI can recognize them when they show up in new environments.

“Frequently, the way we do that is by trying to find pieces of this new scenario that match something that we’ve seen before,” she said. “If we can at least find some landmark pieces that match from what we’ve seen before to what we see now, we can use that to guide our learning on this new source of data.”

In her proposal, Hoffman said her research “will lead to the next generation of vision systems that can operate in novel visual environments with minimal human intervention.” While her focus will be on self-driving cars, she said her research can be utilized in any field that involves image processing, from the medical industry with X-rays and CT scans to autonomous navigation with unmanned aerial vehicles.

Hoffman, who is a member of the Machine Learning Center at Georgia Tech (ML@GT), said the funds from the NSF CAREER Award will allow her to take on new Ph.D. students to assist with her research and grow computer vision research at ML@GT.

While self-driving vehicles appear to be on the doorstep of a breakthrough, Hoffman said scientists should be meticulous about the final steps and not brush over them.

“I think there’s a fair amount of work to do in this direction before it’s going to be put on your phone and integrated tomorrow, or put on a car and integrated,” she said. “I think this work needs to be done to solve those last miles.”