It looks like ProPublica failed its first actuarial exam.

The renowned investigative journalism website has, along with Consumer Reports magazine, published reports that auto insurers systematically charge unfairly high rates to people in minority and low-income communities. (See related article, “ProPublica Report Alleges Auto Insurers Overcharge Minorities,” on Carrier Management’s sister site, Insurance Journal.)

It is an explosive charge—claiming, for example, that in Illinois, 33 out of 34 companies the journalists looked at (including the nation’s largest insurers) all systematically price-gouged minority communities and areas with predominantly low-income households.

And the charge is inaccurate.

ProPublica’s analysis makes an unfair comparison (and a fundamental actuarial mistake) by comparing the losses of all drivers within a ZIP code to the premium charged to a single person.

This is a basic error in statistical analysis, one so elemental that all actuaries learn about it early in their rigorous curriculum. Underwriters learn about it, too. Essentially, the reporters are not making an apples-to-apples comparison.

ProPublica’s investigation is so faulty, it is being criticized not only by U.S. auto insurers but also the state governmental agencies that regulate them.

ProPublica’s journalists claim that bias permeates the industry, all supposedly perpetrated under the nose of state regulators who monitor and often approve in advance the rates insurers charge. And ProPublica said it found discrimination in California, where industry skeptics can earn literally millions of dollars challenging auto insurers’ rate filings.

Discrimination is a charge the insurance industry takes seriously, and in fact, the industry works meticulously with its regulators to ensure that the millions of Americans paying for their products and services are provided fair and accurate pricing. To be clear, insurance companies do not gather information on race or income, nor do they set rates based on this information.

As American society grows increasingly diverse, insurers want to reflect that diversity, both in the people they employ and in the customers they protect. Embracing diversity is good for business. More importantly, it is the right thing to do.

If insurers were to systematically discriminate by race or income, it would be an alarming betrayal of much of what the industry stands for.

The Flaw in ProPublica’s Analysis Explained

Here is why ProPublica’s reporters are wrong: Insurance departments in four states—California, Illinois, Missouri and Texas—provided information on the losses and car counts by ZIP code. The reporters divided the losses paid by the number of cars to calculate what they call a risk factor. (Insurance professionals will recognize that the ratio of loss dollars to exposure is the “pure premium,” or as I will refer to it, the “loss cost.”)

That, the reporters conclude, is evidence of discrimination.

Why is this bad? Well, the loss experience in any ZIP code is driven by all of the factors that make up that area’s drivers. And those drivers together are an eclectic mix. People in some ZIP codes drive, on average, more miles than people elsewhere. People in some ZIP codes get in fewer accidents than people elsewhere.

Here is a super-simplified example to make the point:

Suppose that people who drive “a lot”—say 10,000 miles a year—cost insurance companies $200 in losses per year. And suppose people who drive “a little” (5,000 miles a year) cost insurers $100 in loss payouts. (And assume there are no expenses or taxes. Including them complicates the math here needlessly.)

Further, suppose that there is a ZIP code in which everyone drives a lot and a minority-majority ZIP code in which everyone drives a little. Other than that, drivers are identical in either ZIP code.

How much will it cost to insure a person who drives a lot? $200 in both ZIP codes.

Would the insurer be discriminating? According to ProPublica’s reasoning, the answer is yes.

In the first ZIP code, the ProPublica “risk factor” would be $200 and in the minority-majority ZIP code it would be $100.

And the ratio of premium to risk score—ProPublica’s metric for measuring discrimination—would be 1.0 in the first ZIP code ($200/$200) and 2.0 ($200/$100) in the minority-majority ZIP code. People in the minority-majority area are charged twice as much as the other people, using ProPublica’s flawed logic.

This, according to ProPublica, would be evidence of discrimination. However, the fact is that race or income didn’t create the disparity—it is the fact that the hypothetical driver behaves differently from the typical person in the minority-majority ZIP code.

ProPublica was told about this error—not just once but several times. As chief actuary at the Insurance Information Institute, I explained it to reporter Julia Angwin in a telephone conversation Feb. 13 and in a series of emails dated Feb. 16 and 17. I also know that some of the insurers that stand accused told reporters the same thing.

To its credit, ProPublica shared its data with us. We hired a respected actuarial firm, Pinnacle Actuarial Resources, to independently review the study. They have come to the same conclusion. The journalists had made a rookie mistake.

Because of this mistake, however, ProPublica has concluded that everyone—actuaries, underwriters, executives, regulators—is doing something awful.

And that is why ProPublica journalists found something to accuse insurers of, something that had eluded teams of state regulators, apparently for decades, and scores of industry skeptics.

When you look at what ProPublica did, though, a more likely explanation emerges. ProPublica had a theory—some insurers refused in the 20th century to sell policies to minority consumers—and updated it for a 21st century audience. Its premise today is that, in 2017, insurers are willing to sell policies to minority consumers, but only on the condition that regulators allow insurers to overcharge minorities.

There are other issues I have with ProPublica’s study, including:

• In most states, it doesn’t seem to account for different auto insurance policy limits.
• It doesn’t consider that an auto insurer’s individual loss costs—the amount it pays out in claims—could vary from the statewide averages.
• It doesn’t seem to address how auto insurers priced policies where data about the policyholders and a ZIP code’s loss costs was thin.

These errors could all help produce the result ProPublica reported, but the failure to understand the simple idea of the apples-to-apples comparison is the most obvious error, as well as the least forgivable.

Upon this faulty analysis, ProPublica and Consumer Reports hang a charge of minority discrimination. And they were told, repeatedly, that they were analyzing their data incorrectly. Confronted with this fact, they did nothing, which makes it clear they chose sensationalism over reality.

As a former journalist and a practitioner of the actuarial science, I cannot help but to take this manner of reporting personally. It is irresponsible at the least.