The Base Rate Fallacy

Explaining why we need Bayes in our everyday lives.

Louisa H. Smith

A recent opinion piece in the New York Times introduced the idea of the “Base Rate Fallacy.” We can avoid this fallacy using a fundamental law of probability, Bayes’ theorem. It sounds fancy but we actually already use it to reason in our everyday lives. I’ll motivate it with an example that is analogous to the COVID-19 antibody testing example from the NYT piece.

Headaches and brain tumors

For many people with brain tumors, one of the first signs is a headache. If you have a brain tumor, the probability that you have a headache is quite high. But most of us get headaches, and of course we know that if you have a headache, the probability that you have a brain tumor is quite low, or else we’d all be driving straight to the hospital with every headache. Why don’t we? Because brain tumors are extremely rare.

Similarly, if you’ve had COVID, the probability that you’ll test positive on an antibody test is quite high. In the media, people will say that the accuracy of these tests is great. But what does it mean if you test positive on an antibody test? Unfortunately, it doesn’t necessarily mean you actually have had COVID. Why not? Because – at least so far – COVID is quite rare.

Of course, this depends on the population we’re looking at. We can define “population” any number of ways. For example, if you go to the doctor with a headache, they’ll judge the likelihood that you have a brain tumor, and therefore need more testing, based on the “base rate” of brain tumors in your population. If you live in Chernobyl, it’s more likely that a headache means you have a tumor than if you live in New York, because people with radiation exposure have a higher baseline risk of brain tumors. However, your “population” does not have to be defined geographically. If you have a history of another type of cancer, it’s more likely that a headache means you have a tumor than if you don’t. That’s because people with a history of cancer have the same small risk of a primary brain tumor as everyone else, but also have the additional risk of a metastasized tumor. Their base rate of brain tumors is higher, so a headache is more meaningful.

This is why a positive antibody test carries more meaning in New York than in, say, Maine. If you test positive in New York, it’s more likely you actually had COVID than if you test positive in Maine. And the higher the COVID prevalence in your “population,” defined however you like, the higher the probability that you really had the disease. So if you had symptoms at some point over the past few months, a positive antibody test is more likely to mean that you were actually infected, compared to if you didn’t have symptoms. This is because the “base rate” of COVID is higher among the population of people with symptoms than people without. The same would be true of essential workers, people who have partners who previously tested positive, etc.

We call this the “positive predictive value” (PPV) of a test. Formally, this is the probability that you are positive given that you test positive. When the prevalence of the condition (brain tumor, COVID) in the population is low, all else being equal, the PPV is low.

= Probability of being positive given a positive test
= Pr(are positive | test positive)
= Pr(brain tumor | headache)

(Pr means “probability,” and | means “conditional on,” so you can read that last line “the probability of brain tumor given headache.”)

What about negative tests?

If you don’t have a headache, what’s the likelihood that you don’t have a brain tumor? Extremely high, of course. If you have a negative antibody test, it’s quite likely that you never had COVID. When the prevalence of something is low, a negative test tells you a lot more than a positive test.

We call this the “negative predictive value” (NPV) of a test. This is the probability that you are negative given that you test negative. In other words, the probability that you don’t have a brain tumor given you don’t have a headache, or the probability that you don’t have a history of COVID given that you have a negative antibody test.

= Probability of being negative given a negative test
= Pr(are negative | test negative)
= Pr(no brain tumor | no headache)

So how can we say these tests are accurate?

First of all, “accuracy” is a pretty meaningless word. Since PPV and NPV depend on the prevalence of the condition in the population, we generally judge how “good” tests are based on their sensitivity and specificity. These, like PPV and NPV, are probabilities relating testing positive or negative and actually being positive or negative. However, unlike PPV and NPV, they don’t depend on population prevalence (at least in this very simplified case we’re examining).

In our example, sensitivity is the probability that you will have a headache, given that you have a brain tumor. We said at the outset that’s quite high, so in that one sense a headache is a pretty good brain tumor test. Similarly, if you have indeed had COVID, you’re quite likely to have positive antibody test. In other words, how good is the test at picking up (sensing) this condition?

= Probability of testing positive given being positive
= Pr(test positive | are positive)
= Pr(headache | brain tumor)

Specificity is the same idea with respect to testing and being negative. If you don’t have a brain tumor, what’s the probability you don’t have a headache. (Not super high – there are a lot of other reasons for a headache! A headache is not a good test in this sense.)

= Probability of testing negative given being negative
= Pr(test negative | are negative)
= Pr(no headache | no brain tumor)

Similarly, If you don’t have a history of COVID, what’s the probability you test negative on the antibody test. Quite high! A test that is good at ruling out a condition is highly specific.

So the COVID antibody tests generally meet both of our criteria for being a “good” test (without putting numbers to these values, since they depend on the test and other factors – it’s more complicated than I’ve made it out to be!). However, because many, many more people are truly negative for COVID than positive (the prevalence is low), a positive test result does not necessarily mean you have actually been infected.

So what is Bayes’ theorem?

Bayes’ theorem tells us how these probabilities relate, given the prevalence of the condition.

For events A and B, Bayes’ theorem is

\[ \Pr(A \mid B) = \frac{\Pr(B \mid A)\times \Pr(A)}{\Pr(B)} \]

which we can also write

\[ \Pr(A \mid B) = \]\[ \frac{\Pr(B \mid A)\times \Pr(A)}{\Pr(B \mid A)\times \Pr(A) + \Pr(B \mid not \;A)\times \Pr(not\; A)} \]

What does this mean in terms of our examples? Well if the events of interest are “being positive” and “testing positive,” we can write this

\[ \Pr(\text{are positive} \mid \text{test positive}) = \]\[ \frac{\Pr(\text{test pos} \mid \text{are pos}) \times \Pr(\text{are pos})}{\Pr(\text{test pos} | \text{are pos})\Pr(\text{are pos}) + \Pr(\text{test pos} | \text{are neg})\Pr(\text{are neg})} \]

Then, using the above definitions and the fact that probabilities must add up to 1, we have

\[ \text{PPV} = \]\[ \frac{\text{sensitivity} \times \text{prev}}{\text{sensitivity} \times \text{prev} + (1 - \text{specificity}) \times (1 - \text{prev})} \]

Now assume that sensitivity is 1: everyone with a brain tumor has a headache, or the test picks up on all the true COVID infections. This isn’t the case, but makes things easier and doesn’t really matter for our purposes. Then we have that

\[ \text{PPV} = \]\[ \frac{\text{prev}}{\text{prev} + (1 - \text{spec}) \times (1 - \text{prev})} \]

Now let’s say that specificity is 0.95. In other words, 95% of people who never have COVID test negative (the other 5% are false positives). If prevalence is 1% (Maine, maybe) vs. 20% (New York, maybe), we have PPVs of \(\frac{.01}{.01 + .05 \times .99} = 0.17\) and \(\frac{.20}{.20 + .05 \times .80} = 0.83\), respectively. That means a positive test in Maine means that you only have 17% chance of truly having had the disease, but a positive test in New York gives you an 83% chance!

Notice also that if specificity is 1, no matter the sensitivity or the prevalence, the PPV is also 1! Since there are no false positives, any positive test is a definitive positive. Unfortunately, that’s not true of many types of tests.

You can use the tool below to play around the the values to see what a positive or negative test means in various situations:

But why isn’t the test correct?

There are a number of reasons why a test might not be right. If you’re truly negative, it may pick up similar antibodies for another virus, it may be contaminated in the lab, it may be misread by a lab tech – any of these and more could lead to a false positive. If you’re truly positive, being tested too soon after infection, as well as various lab errors, could also lead to a false negative.

There are testing strategies to maximize the predictive values of tests. Tests can be conducted only in high prevalence populations, whether that’s defined by symptoms, exposure, or geographically. Repeat testing after a positive test can also increase the PPV. Tests with different properties can be used simultaneously to reduce error.

We encounter these concepts in everyday life, even if we’re not directly thinking about Bayes’ theorem. In order to reduce false positives, screening mammograms are only given in older women, as they have a much higher base rate of breast cancer than younger women. After a positive mammogram, repeat imaging and then a biopsy are subsequent tests used to rule out false positives. As COVID antibody testing becomes more widely available and new tests are developed, we can apply these principles in laying out a testing strategy. Importantly (and unfortunately), a positive test result does not currently mean that we can give up on social distancing and other precautions.

The CDC’s current guidelines on antibody testing are available here.