In many fields people try to predict group membership. One example includes making diagnostic predictions (e.g., does this person have a disease or diagnosis?). Another example is predicting who will attempt a terrorist act. Malcolm Gladwell, in the New Yorker, has an excellent article on this topic. One difficulty that he and Meehl and Rosen (1955) point out is the base rate problem.
For example, we could ask: What is the probability that people have panic disorder given our new measure of panic disorder identifying them as having panic disorder? Let’s start with two extreme examples. If no one in our sample has panic disorder (the probability is zero; p=0.0), then there are no people to find with panic disorder and we are wasting people’s time (anyone we identify with panic disorder actually does not have it, so we are wrong when we identify them as having it. If everyone in our sample has panic disorder (the probability is one; p=1.0) then everyone has panic disorder and our test is great at identifying anyone with panic disorder (we have high accuracy because we can’t go wrong), but we do terribly when we miss people. In both of those examples, our test could be a great test or an awful test, but it does not have great utility because of the extremeness of the samples.
A similar problem exists when we try to identify people who want to conduct a terrorist act. There are very few of these people in the population (so probability is close to zero). Thus, we incorrectly identify many people who aren’t terrorists in the hope of finding those few people whom are.
The mathematics of these probabilities were figured out by the Scottish logician Thomas Bayes. Bayes’ theorem requires three estimations:
1. The base-rate for panic disorder (we’ll continue with the same panic disorder example) i.e. what proportion of the population taking the test have panic disorder;
2. The accuracy rate of the test, i.e., the probability that people with real panic disorder will be identified as having panic disorder by our test;
3. The misidentification rate of the test, i.e., the probability that people without panic disorder will be misidentified by our test as having panic disorder.
Figure 1. General Formulas for Diagnostics
If we were to use our new test for panic disorder on the general population (which probably has a base rate of 3 to 10% of people having panic disorder) we will do much worse at identifying panic disorder than if we test people in out specialty panic clinic (where everyone walking through the door thinks they have panic disorder. Thus, the population at this clinic probably has a much higher rate of panic disorder). The above figures illustrate this concept by using a hypothetical test with a sensitivity (identifying people who have the disorder) of .85 and a specificity (identifying people who do not have the disorder) of .89 (these numbers would indicate that we have a pretty good test).
By keeping the same test, which has the same sensitivity and specificity of finding panic disorder and changing the sample of people we test, we find greatly different results. Figure 2 shows the sample of people coming into our specialty panic clinic. 96% of these people actually have panic disorder. Figure 3 shows the general population with a prevalence of 5% (5% of the people in this sample have panic disorder). In the general population sample, we incorrectly identify 10 people as having panic disorder, when indeed they do not (although, we do correctly identify 4 out of the 5 people with panic disorder as having panic disorder). In the panic clinic sample, we miss identifying 14 people who actually have panic disorder. Thus, the same test has very different results in different populations.
Figure 2. 96% Prevalence Rate
Figure 3. 5% Prevalence Rate