Recently, Cruise announced our safety record after completing our first million fully driverless miles. We compared our driverless safety performance to an approximate human benchmark, which we offered as a refinement of the frequently-cited U.S. national average statistic produced by NHTSA¹. Today, we are presenting the foundational research underlying the human benchmark in a white paper published by our lead research partners at the University of Michigan Transportation Research Institute (UMTRI). Together with General Motors and the Virginia Tech Transportation Institute (VTTI), UMTRI conducted an unprecedented large-scale naturalistic study consisting of 5.6 million miles² of human ridehail driving data to measure human driving performance in a targeted urban environment. Ultimately, the white paper calculated a human driver crash rate that Cruise used as our foundational benchmark to measure the safety performance of our driverless fleet. 

“What’s been missing in autonomous vehicle research is a benchmark that goes beyond the available data and accurately reflects the human driver. With support from General Motors and Cruise, experts at UMTRI were able to take a groundbreaking new approach to generating human-driver benchmarks in environments comparable to ADS deployments,” said Dr. Carol Flannagan, lead author of the paper and research professor at UMTRI. “We investigated actual driving behaviors of ridehail drivers in a complex urban environment that can be meaningfully utilized as a benchmark for comparable human driver performance.”

The White Paper, Summarized

The data collection efforts of this study took place over a two-year period from 2016 to 2018, with extensive collaboration between Cruise, General Motors (GM), UMTRI and VTTI – two of the leading transportation research centers in the United States. The study’s Operational Design Domain (ODD) included the entirety of the city of San Francisco, excluding select high speed roads (e.g. posted speeds of greater than 35 mph). Additionally, the study measured the performance of human ridehail drivers.

Using two concurrent naturalistic driving studies (UMTRI’s large-scale study, and VTTI’s precision-instrument study, as well as a shared UMTRI-VTTI fleet) allowed the research team to derive a statistically robust measurement of the human ridehail crash rate in the designated ODD. The larger UMTRI fleet was owned by the Maven subsidiary of GM, and provided a larger dataset of miles and events, whereas the VTTI fleet was partially owned by the Maven subsidiary and partially personally owned by the drivers, and provided a smaller but more precise dataset of miles and events. The shared fleet between the two studies allowed for a detailed comparison of the respective crash-detection mechanisms.

Here is a summary of the data output from each fleet:

Table 1 data available here

The total mileage of UMTRI Fleet and VTTI Fleet³ produced a total of 5,611,765 ODD miles observed in this study.

Use of a Bayesian Fusion statistical model⁴ generated the human ridehail crash rate:

1 crash in 15,414.4 ODD driving miles, or 64.9 crashes per million ODD miles

Analyzing the Human Ridehail Crash Rate

The human ridehail crash rate benchmark developed with UMTRI and VTTI data in the white paper addresses the overall crash rate, or frequency of crashes, of human ridehail driving in San Francisco. But it did not address meaningful risk of injury or primary contribution.

To expand on these elements, Cruise performed additional analysis of the benchmarking data to develop metrics for meaningful risk of injury and primary contribution. When we think about individual crashes, we not only consider that a crash took place; but we also consider how much each party was responsible for the crash, as well as how severe the crash was. A fender bender at a stop sign has different implications on the measurement of driving behavior than a high-speed blow through of a red light. Identifying subsets of the overall benchmark, such as primary contribution and meaningful risk of injury, allow us to better understand the meaning behind the crashes. 

Meaningful Risk of Injury

Identifying crashes with meaningful risk of injury allows us to more specifically characterize the driving performance of the human ridehail crash rate based on whether a given crash is likely to result in injury. Crashes that take place with higher delta-velocity (dV) have a strong relationship to risk of injury, when compared to crashes that take place at lower delta-velocity. Crashes with meaningful risk of injury do not always result in actual injury, but are noted as safety events.

The table below lists the crash categories observed by UMTRI and VTTI according to severity.

Table 2 data available here

Given that Level 3 crashes are characterized by VTTI as “minor” and UMTRI as “minimal risk,” we have defined crashes with meaningful risk of injury as crashes that measure as Level 2 or more. 

There were 60 L2+ crashes observed in the UMTRI Fleet and 6 L2+ crashes observed in the VTTI fleet, which produced an estimate of 66 L2+ ODD crashes in a total of 5,611,763 ODD miles.  Unlike minor L3 crashes, which may have been underreported and for which there was discrepancy between the two fleets as explained in the white paper, neither fleet detected a discrepancy for the more severe L2+ crashes, which gives us confidence that there were no missed L2+ crashes during the study. 

This estimate produces a human ridehail crash rate with meaningful risk of injury (L2+) in ODD of 1 crash in 85,027 miles; or, 11.76 L2+ ODD crashes per million miles.

Primary Contribution

According to the motor-vehicle crash outcomes in the U.S. in 2021, there were a total of 13,200,000 crashes with 23,600,000 unique vehicles involved.⁷ This produces an estimation of 1.8 vehicles to 1 crash event, which indicates that most crashes on the road in the U.S. involve an average of two vehicles. We further reviewed the available human ridehail crash data, which showed a similar rate of vehicle to collisions and slightly greater than 50% responsibility for the VTTI subset of crashes.

Given this estimation, and the principle that one party is generally found to be primarily responsible for a given two-vehicle crash (i.e. found to have primary contribution), we assert that a 50% distribution of primary contribution to the average human driver who is involved in a crash is a reasonable assumption. 

By applying this assumption to the human ridehail crash benchmark established above, we derive a coarse estimate of 1 crash in 30,828.8 ODD miles (i.e., double the mileage per crash); or, 32.45 crashes per million miles (i.e., half the number of crashes).

Collective Human Ridehail Crash Rate Benchmark

Based on the white paper, and the assertions above, the human ridehail crash rate benchmark for San Francisco is as follows:

Table 3 data available here

Note that this white paper offers a more refined estimate of the human benchmark performance compared to the coarse estimates published in our first million driverless miles. These refined values provide greater granularity in our measurement of the human benchmark.

Measuring Safety Performance

As we continue to collect driverless miles and provide updates of our driverless safety record, we want to set a meaningful and reasonable human benchmark for measuring our safety performance. This study is the best available benchmark for measuring human driving performance in a dense urban environment. We carefully collaborated this white paper to ensure full transparency regarding our analysis and to demonstrate our confidence in the data we published as our safety benchmark in San Francisco. By publishing this blog, we hope to make our safety methodology, the crash rate benchmark developed with UMTRI and VTTI, and Cruise’s augmentations for primary contribution and meaningful risk of injury, clear and readily understood. We will use the precise human ridehail benchmark for updating our safety record moving forward, and encourage the industry to adopt a similar standard of tangible safety performance.