Probabilities are Conditional and That’s a Bummer
There is so much frustration with governors for locking down, and with the federal government for not having enough test kits. It feels as if we’re not taking enough action, moving fast enough on approving and manufacturing antibody test kits and actually testing people. I’ve been advocating for widespread antibody testing once they were available. It seemed to make sense as the logical way to find out who can get back to work and move our world forward again. It felt rather obvious to me that if we had a test for antibodies, that we should be able to apply this test to everyone as soon as possible. It seemed to reason that if a significant number of people already have had Covid-19 then they must already have immunity. Testing for antibodies is supposed to give you an indication that you’ve had the virus and are therefore are immune. The prevailing logic, including my own, was that if we could test people and see that they test positive for the antibodies, that they could go back to work.
I think we have a lot of smart, adept people on the case, in spite of what we may think and hear from our social media friends, pundits and the media. So, I haven’t let this one go and have continued to sharpen the pencil. After some deeper statistical analysis and attaching some common sense to it as well, I understand better now why the CDC would not advocate widespread antibody testing. It turns out, antibody tests might only be right as low as 20% of the time and probably as high as 53% of the time right now. Would it make sense to send people back to work based on a test that’s only right 20-50% of the time? You see, it really matters how and where you apply the test. The problem becomes tied to two major variables.
The first is the percentage of people in our population that have had the virus. Let’s say that 1% of the U.S. population has had Covid-19 and would therefore have the antibodies. That would be roughly 3.3M people in the U.S. If 5% of the population has it, then we would have 16.5 M with the antibodies. Taking it further, you can see that 50% of the population would be roughly 165M people. This becomes very important to the usefulness of testing everyone as you’ll see. Today as it stands, we are not sure what this number actually is. It seems like that’s the whole purpose of testing a lot of people right? But ironically, this number, the actual number of people who have had the virus, can radically effect the accuracy of the antibody test results.
The second critical number is the % of people who don’t have the antibodies but show up as positive on the test anyway. These are referred to as “false positives”
Let’s start with these tests themselves and how they are measured for effectiveness. Cellex has the only FDA approved antibody test kit available for testing Covid-19 antibodies in the U.S. There are two measures that indicate its effectiveness. It has what’s called a sensitivity of 93.5% and a specificity of 96.8%.
Here’s what that means with respect to this antibody test.
Sensitivity - measures the probability that a test will identify someone positive, meaning that the test will identify that someone has been exposed and therefore has antibodies as positive This number was listed at 93.8% That means if 1,000 antibody carrying people are tested, this test will correctly identify 938 of them as testing positive for the antibodies, but will also show that 62 of them test negative.
Specificity - measures the probability that a test will accurately identify someone as negative, meaning they don’t have the antibodies given that they in fact don’t have them. This number was listed as 95.6%. That means that if 1,000 people are tested that don’t have the antibodies at all, in fact they are all negative, then the test will correctly show 956 of them as negative, but it will also falsely show that 44 of them DO have the antibodies. Another way to say this, is that the test produces 44 false positives. This is the important distinction. This number is what will mess with your head when you apply it to the population at large.
Combined Effect - now let’s add these two numbers together and assume a population where 1,000 people have the antibodies, and 1,000 people do not. In other words, this is a population with 50% of the people having had the virus (we are likely at less). Based on this sensitivity and specificity, this test would show 938 people accurately have the antibodies, and 44 people who falsely have the antibodies. In this example, the test shows a total of 982 people as having the antibodies. If we complete the new math, then 938/982, or 95.5% actually have the antibodies and 4.5% do not.
This is example looks great! The actual effect is even better than the sensitivity listed above. We should be thrilled. But there’s a problem, its heavily dependent upon the percentage of the population that actually has the antibodies.
Do we have 50% of our population infected? As of this writing, we have over 525K cases in the US. Which is roughly 0.15% of the population. Our instincts seem to dictate that way more people than that have had the virus. But what’s the number? You’ll see that if we don’t know, then it’s hard to accurately use the test results. For example, if we thought that 16.5 M people actually have had the disease, then that would represent roughly 5% of the population.
Let’s use the sensitivity and specificity of Cellex as listed above and apply them to that 5% example I just mentioned. After all, we want to test as many people, so we can mark them as having the antibodies and send them back to work right? Well, this thinking breaks down with conditional probabilities, and when applying Bayes Theorem to problem. I won’t go through the statistical approach (happy to do so with you directly). Instead, I'll display its logic visually below.
Widespread testing using antibody tests MAY NOT yield value yet
One of the prevailing thoughts as mentioned at the beginning was that we could test all 330 Million people in the U.S. to figure out right now who has antibodies and who does not to inform who can return to work. The figure breaks this down as follows
% of Population: shows the population in percentage and raw numbers of people who have and don’t have antibodies.
Test Accuracy: applies the sensitivity measure to the population groups.
Antibody - this population has the test positive percentage applied to it (sensitivity) of 93.8%
No Antibody - this population has the false positive percentage applied to it (1-specificity) 4.4%
Test Output: shows the numerical impact of these percentages
Antibody - if we correctly identify 93.8% of actual positive people with antibodies, we’’ll get 15.5 Million people with the antibodies
No Antibody - if we take the false positive rate and apply it to this much larger population, we get 13.1 Million people inaccurately identified as having the antibodies
Result:
If you take 15.5 Million actually positive + 13.1 Million Falsely positive you have 28.6 Million people identified as having the antibodies across the entire population.
If you take the 15.5 Million who actually have the antibodies / 28.6 Million in total identified as positive, you get 53%.
In other words there’s a 53% chance that if you take a test and it shows you have the antibodies, that you actually in fact do have them.
Is that good enough to take an action? Would you go back to work knowing your antibody test has a 50/50 chance of being right? Moreover, if it’s wrong, and you go back to work, interacting with other people, you might get infected. If you’re infected, with a 2-3 week incubation period, you might just be out there spreading new virus to everyone you interact with. It’s hard to see a world where social distancing and precautionary measures are disappearing anytime soon. Antibody tests don’t seem like they will be effective in accelerating the inputs to our models and allowing us to put people back to work until we have a much higher confidence in the percentage of people that have the antibodies.
You see, the test kit accuracy numbers are great and line up just fine when you are testing sick people. When you are testing a much bigger population, most of whom do not have the antibodies, that specificity number, the one that inaccurately states people have the antibodies when they don’t, can bite you. Now all of a sudden, when applied to the entire population, the number of these “false positives” becomes highly problematic. Since we’re using a positive to indicate that you have the antibodies and are safe, we’d suddenly begin marking a lot of “unsafe” people as “safe”
Tests vary based on population infected and specificity
Probability that a “positive” test is accurate given various levels of infection in the population at 95.6% specificity
Each column represents a % of the population that ACTUALLY has Covid-19 Antibodies
The values in each column in the bolded box show the probability that if you test positive for antibodies, that you ACTUALLY have those antibodies.
It's so easy to think the CDC is being too slow, or not testing enough people, but perhaps we can also give them some benefit of the doubt that they know what they are doing. There continues to be no silver bullet here. But that doesn’t mean we can’t keep trying. We can improve the specificity of the tests. That can certainly helps change the outcome.
Here’s the same table at 99% specificity.
Each column represents a % of the population that ACTUALLY has Covid-19 Antibodies
The values in each column in the bolded box show the probability that if you test positive for antibodies, that you ACTUALLY have those antibodies.
Improving the specificity improves the outcome if 1% of the population has the disease, but it now matches the 5% case in the original example. Its better, but still not good enough to make me feel confident about it. So what’s the answer? Improving sensitivity certainly won’t hurt. Will a second test make a difference? It seems unlikely that a second test of the same kind will improve the probability of an accurate outcome. Whatever the reasons are that your chemistry or physiology is triggering a false positive may happen again. I’m left with thinking that if there were a second test of a different type or method, we might do better on triangulating the true value of the test.
I posted earlier about how testing is the most important point in the infection funnel. While i’m suggesting our current approach has gaps, we need conscientious study of all aspects of these applications. We need to improve specificity, apply multiple test types, and continuously and accurately triangulate what percentage of the population actually has the disease. My pediatrician recently announced the existence of an antibody test that’s available for patients to take. Knowing now what I know, I realize that its probably not worth it yet nor is it particularly helpful at this stage. We need a thoughtful, clearly communicated approach to all stakeholders involved on what to do with these tests, how many to use, how to use them, and what actions we can take based on the information they give us. We need them to work, we’re just not there yet.
Sources
http://www.centerforhealthsecurity.org/resources/COVID-19/serology/Serology-based-tests-for-COVID-19.html
https://www.ncbi.nlm.nih.gov/pubmed/26651986
https://en.wikipedia.org/wiki/Bayes%27_theorem
http://vassarstats.net/clin2.html