NHTSA Claim That Tesla Autosteer Reduced Crashes Proven Bogus

In January 2017, the National Highway Traffic Safety Administration released a statement on the safety and efficacy of Tesla’s Autosteer technology. According to the Office of Defects Investigation (ODI), Tesla’s lane-keeping technology had reduced accident rates by nearly 40 percent. We now know that number was completely wrong, based on a new analysis of the same data set the NHTSA used to reach that conclusion.

The analysis firm Quality Control Systems filed an FOIA request with the NHTSA to see the actual data the organization had used. The NHTSA had already backed away from its own conclusion, having issued a statement a bit less than a year ago that the 40-percent figure was merely a preliminary conclusion. What QCS found upon investigation, however, was a set of errors so egregious, they wrecked any predictive capability that could be drawn from the data set at all. In fact, the most narrow read of the most accurate data set would suggest that enabling Autosteer actually increased the rate of Tesla accidents by 59 percent.

Or rather, it did if the data is actually accurate. And that’s not clear.

The NHTSA originally reported its findings as a comparison between crash rates before and after Autosteer was activated. This crash rate data was calculated by comparing the number of airbag inflations per distance driven both before and after Autosteer was available. But this only works if the total number of miles driven before and after Autosteer are drawn from the same pool of vehicles. And based on QCS’ examination of the data Tesla provided to the NHTSA, only 5,714 vehicles met that criteria out of 43,781 vehicles included in the data set.

The problem lies in the information Tesla provided. The company gave the NHTSA the last odometer reading it had on file before Autosteer was installed and the first odometer reading after Autosteer was activated. In 5,714 cars, this was the same number — meaning we know these vehicles were tracked both before and after Autosteer.

The entire rest of the data set is missing data in one way or another. Some vehicles only reported mileage after Autosteer had been installed. Some reported mileage before and after installation, but not when the installation occurred, making it impossible to know whether accidents in-between the two data points had occurred with Autosteer installed or not.

In the figure above, the NHTSA assigned no exposure miles (traveled distance) to 14,260 vehicles in its data set, yet did assign 15 airbag deployments to this same segment. This is self-evidently bad math. If the goal is to measure the number of airbag deployments per mile, 15 airbag deployments in zero miles traveled will obviously make Tesla’s pre-Autosteer crash rate look much too high.

The authors don’t speculate much about why the small number of vehicles with the most complete data set show the 59 percent increase in Tesla crashes once Autosteer was enabled, but there’s some reason to think that this data could be incorrect as well. The entire data set is bad enough that the report authors aren’t comfortable declaring how Autosteer definitively impacted vehicle safety.

At a minimum, the NHTSA data was badly analyzed and the organization should never have released the report it gave. Interestingly enough, the reason it took so long for us to find out about all this is that once QCS started investigating, Tesla tried to argue that the same data it had voluntarily provided to the NHTSA for analysis constituted a trade secret that couldn’t be discussed in public. The NHTSA parroted this back to the courts, forcing QCS to sue for the information. In the meantime, a bad data point that should’ve never been issued became an industry talking point and established “fact” with no basis in empirical reality.

More than two years after Tesla released Autosteer we still don’t know if these technologies are statistically improving vehicle safety or not. That doesn’t mean they aren’t — but the data that would make that case has yet to be shown.

Continue reading

NHTSA Opens Probe into Tesla Gaming While Driving

It’s currently unclear whether Tesla considers the ability to game and drive to be a feature or a bug . . . but a few clues point toward the former.