When cars sit in traffic or at stoplights, they waste gas. Larger vehicles like tractor trailers use more fuel than cars when they idle and when they’re getting back up to speed after stopping at a red light. In fact, the Department of Energy estimates that more than 6 billion gallons of gasoline and diesel combined are lost to idling every year.

Researchers at Oak Ridge National Laboratory (ORNL) used artificial intelligence and machine learning to design a computer vision system that helps keep traffic moving efficiently through intersections and has the added benefit of minimizing fuel consumption.

The team used three smart stoplight cameras from Gridsmart, a traffic management services company, to collect real-world data from images of vehicles as they traveled through intersections on the lab’s campus. They applied AI and ML to teach the cameras how to identify vehicle types and their estimated gas mileage and then send that data to the next intersection’s traffic light.

Because the Gridsmart cameras only supply an overhead view – rather than a ground view — of vehicles, the researchers tapped a ground-based roadside sensor ORNL was using for another project so they could collect data on vehicle makes and models. They the overhead images and ground-based data to get a dataset of vehicle classes. When combined with DOE fuel-economy estimates for different vehicle types, the ORNL Overhead Vehicle Dataset would allow them to determine when gas-guzzlers were in the intersection.

“The basic gist of it is: If you can use their imagery to detect the make and model, you have a pretty good estimate of what the vehicle’s fuel consumption is,” Karnowski said. “Then the second part is [using] that information to control the traffic lights to save energy without costing mobility,” he said. “We did some simulations based on the datasets we collected to see if we could answer that question.”

Between the yearlong project’s start in February 2018 and September 2018, the dataset grew to about 12,600 vehicles spanning 474 classifications, but Karnowski said that wasn’t enough to train a deep learning network that would automatically detect and classify vehicle types. (The project received a six-month extension in February 2019.) Without the time to collect more data on their own, the researchers turned to a publicly available dataset collected by Stanford University researcher Timnit Gebru that identified 22 million cars from Google Street View images and classified them into 2,600-plus categories.

The team lowered the resolution on Gebru’s dataset to make it equivalent to what they got from Gridsmart and used the AlexNet convolutional neural network to start the project’s second phase. They tweaked Gebru data to estimate fuel consumption by substituting vehicle types with the DOE fuel consumption averages and got results similar to those they got from the Gridsmart data.

The researchers then built another neural network to compare the datasets using ORNL’s Multinode Evolutionary Neural Networks for Deep Learning (MENNDL) high-performance computing software stack.

“We got results that were essentially comparable to what we had with the more conventional neural network, but I believe if we had had more time and more data, we could have gotten better results because MENNDL has done a good job on a lot of other problems before,” Karnowski said. “That did allow us to get a sense of, informed by real-world data, how well these cameras could be trained to detect the vehicle make and model.”

To run a simulation of a citywide traffic grid, Karnowski and his team used an open source package called SUMO to model traffic systems. They then added the ability to train an ML algorithm to affect the lights with a detection system that was informed by their experimentation with the Gridsmart and open source data. They used an ML approach called reinforcement learning to teach the system to change the traffic lights to keep high fuel consumption vehicles moving.

“What’s interesting about it is you basically set up a system of rewards and penalties, and then you let the computer try different things until it learns to get the rewards and minimize the penalties,” like learning to win at a game, Karnowski said. “You simply say, ‘You make this move and you see what happens, you make that move and you see what happens.’ By the time it gets to the end, if it wins, it learns those moves were good,” he explained. “Over time, over thousands and millions of iterations of simulations, it learns how to play the game really well.”

In this case, the game was training the traffic light to keep traffic moving and save energy at the same time. Although the team didn’t do a complete round of simulations, Karnowski said, in at least one scenario, they estimated a 25% fuel savings in the case of large trucks moving in one direction.

“It learned how to preferentially treat the large trucks because they are heavy fuel consumers,” he said, without simply leaving a lane light green for longer. “It was actually actively changing the traffic signal based on the traffic that it saw, the makeup of the traffic and then other considerations, like you don’t want to have people waiting too long, so you have to keep that in mind as you control the light,” he said.

The project, which was funded by HPC4Mobility, the Energy Department’s Vehicle Technologies Office’s program for studying efficiency in mobility systems, has ended, but Karnowski said he hopes that with the proof of concept his team established, municipalities can move forward with more studies.