Lies, Damned Lies, and Vaccine Statistics

Hyper-vigilant vaccination advocates are pushing dangerous misinformation about vaccine efficacy

Accurate description of effects is not hair-splitting; it is much-needed exactness to avoid adding confusion to an extraordinarily complicated and tense scientific and societal debate around COVID-19 vaccines.

It does not mean that 95% of people are protected from disease with the vaccine—a general misconception of vaccine protection also found in a Lancet Infectious Diseases Editorial.

This appeared in correspondence in a March 8th publication through The Lancet, a leading peer reviewed medical journal. It emphasizes that even within the scientific communities focused on such issues, misunderstandings about the meaning and implications of reported numbers about vaccine efficacy persist. If the scientific community makes a mistake in its own interpretations, that can misinform policy makers, even very educated ones, and that in turn can misinform the public.

In the last few weeks I have seen evidence of articles more widely shared to correct some of the early misconceptions The Lancet correspondence was concerned about. However a more insidious form of misunderstanding and misinformation lingers, and is pervasive.

Dr. Tom Frieden @DrTomFrieden

It's actually pretty simple math. If you get Covid and you've been vaccinated, you're about 100 times less likely to die.

Tom is no ordinary doctor. He is the former Director of the Center for Disease Control under President Barack Obama. He is also propagating dangerous misinformation about the COVID-19 vaccines. It will cause people to misunderstand the real world results which can cause more people to die, and exacerbate the pandemic.

Dr. Tom is wrong. About as wrong you can get. The real world data has shown that the death rate among the vaccinated, if infected with COVID, can be 3 to 5.7 times higher¹ than the death rate of the unvaccinated.

Acceptable Catastrophic Error

The former Director of the CDC is making what I call an Acceptable Catastrophic Error. This is the kind of error one is allowed to make when they are perceived to have the correct opinion. Arguments and statistics used towards the goal of getting every single person vaccinated for COVID are given far less scrutiny and are accepted as true more readily, than any arguments or statistics that might be perceived as counterproductive towards that goal.

When the former Director of the CDC, or anyone else for that matter, says someone who is infected with COVID is 100 times less likely to die if they have been vaccinated, it does not matter how far off that number is. It will be readily forgiven no matter how inaccurate. It is the goal of universal vaccination, not truth, that appears to be most important.

There are a lot of acceptable catastrophic errors.

Vaccine Efficacy

In order to understand why people are claiming these marvelous additional benefits exist, we will unfortunately have to do some math. But once we do, confusion will be alleviated, and the dangerous misinformation being spread by those in authority and those on a mission to convince everyone to get vaccinated, will become clear.

To understand vaccine efficacy, a diagram provided in the results of a Pfizer–BioNTech mRNA BNT162b2 clinical trial will be very useful.

The red line with circles shows that for those who received the vaccine, which was 21,669 people, the rate of infection slowed. The average number of days between individual infections increased. The result is that over a period of time, the total number of people who became infected was indeed far lower than the unvaccinated group.

This is where the earlier statement from from The Lancet “who would have become diseased with COVID-19 had they not been vaccinated” comes into play. It is reasonably assumed that the red line would resemble the blue line if the group were unvaccinated. Therefore one could count the number of squares on the blue line that occur in between instances of red circles, and get a ratio of how many unvaccinated people got infected per each vaccinated person over a period of time.

The efficacy rating (VE in the table) is a way of trying to do this. However one additional layer of complexity is added. Researchers treat the people in study without much distinction. Every day that a person is included in the study and observable is considered a person-day. Every person being observed on a given day is counted and added to a tally of person-days. At the end of the study, the observed infections are divided by the total person-days counted for the group.

For example, if two people are observed across four days, it is a total of eight person-days. Each person has a chance of infection each day, unless they were already infected. It makes a total of eight chances for a person to become infected. However, once infected, someone cannot be infected again, so that person stops being counted in the person-day tally. In this particular case they used a different length of time than day, but the idea is the same.

At the end of the day, the efficacy calculation is the result of counting things and dividing things counted. We can take the numbers they counted and divide them ourselves to arrive at the same efficacy numbers. Let’s use the data from ≥ 7 days after dose 2. We have 21,314 people across 4.015 time units for the vaccinated group, and 21,258 people across 3.982 time units for the unvaccinated. Now we compare the rates of infection, which were 9 and 172 observed infections respectively.

Finally, we take the two rates, and compare them, and compute a number that represents the percent score called efficacy.

There is the efficacy number from the table. Subtracting from 1 simply allows you to report a number where higher is better, instead of using “golf rules” where lower is better. The calculation itself is not misleading if you understand how it is derived. But it is important to hold on to the idea we are currently talking about infection rates over time only. It is perfectly reasonable to compare infection rates among two groups in this way.

This distinction is all the more important as, although we know the risk reduction achieved by these vaccines under trial conditions, we do not know whether and how it could vary if the vaccines were deployed on populations with different exposures, transmission levels, and attack rates.²

Real populations

Israel offered a glimpse into the real world dynamics of the Pfizer–BioNTech mRNA BNT162b2 vaccine. A major study, published May 2021 in The Lancet, reviewed national surveillance data, collected as part of a government vaccination initiative. The data collected was able to identify those vaccinated on which dates; when they were diagnosed with COVID; whether they were symptomatic or not; if and when they went to the hospital; if they had severe or critical hospitalization; and if they died.

The study is quite thorough, although it did contain errors some of which may not have been fully corrected. But we can jump directly to inspecting the efficacy numbers and correcting errors. We will be using the numbers provided in Table 4 on page 1825.

The top white box contains the number of infections observed during the study. We can use All Ages. The Unvaccinated and Vaccinated infections for All Ages were 109,876 and 3,642 respectively. Just as before, we need the person-days (the total time observed) for both. These are provided at the bottom of the table and for the Unvaccinated and Vaccinated they are 120,076,136 and 170,434,659 respectively.

Taking a look at our approximate ratios calculated, you can see that 91.5 in the red square in the table corresponds to our calculated unvaccinated rate, and similarly 2.1 in the blue box corresponds to our calculated vaccinated rate. We would simply need to multiply both by 100,000, as they are incidence rates per 100,000 person-days, to make them about equal to the table.

The next step is identical to the Pfizer phase three trial. “Vaccine effectiveness estimates were calculated as (1 – IRR) × 100” (Incidence Rate Ratios, IRR)

Our calculation appears to be 1.1% higher than that in the table. There are two possible explanations. One explanation may be that when they corrected some hard values they failed to update the efficacy calculation. The second explanation is that the authors ran the raw data, which we do not have, through a particular statistical package³ in order to estimate the bounds (numbers in parentheses, giving some wiggle room to the estimate), which gave them a more nuanced estimate for the central incidence rates than the numbers they provide the reader allow — as our calculated incidence rates are clearly in agreement with the table.

This isn’t too important, as these are simply estimates the authors are making based on observations. These estimates will change over time and with different populations. They are very useful to have, but should not be treated as if they’re exact.

They are also not where the major issues arise. One can simply look at the raw numbers without calculating the efficacy number and conclude that indeed, infection rates were substantially lower. The differences that remain in their calculation for efficacy percentage are negligible.

Next, however, we move to what will ultimately be the source of misinformation. It isn’t the fault of the report itself, but a consequence of too many people who pretend to speak from knowledge and authority interpreting numbers they do not understand.

Vaccine Efficacy “against death”

Included in the study are rates of symptomatic COVID, hospitalization, severe hospitalization, and their respective efficacy calculations. At the bottom of the table are efficacy calculations related to death. We will use the table to make one final efficacy calculation so we can move to discussing this horrific source of misinformation.

We are given 36 deaths among those aged 16-44 in the unvaccinated group and 0 deaths in the vaccinated group. We have 88,938,310 and 61,397,072 person days for unvaccinated and vaccinated respectively from the bottom of the table. This gives us, indeed 100% efficacy!

100% Efficacy! That’s 100% protection against death! Those between the ages 16-44 cannot die from COVID! You can see how impossible it is for someone to die if vaccinated, as they do not even provide a range of uncertainty. 100% with no uncertainty is as certain as you can get.

The Fatal Flaw (pun intended)

It is obviously possible for vaccinated people 16-44 to die from COVID. The other results from this efficacy from death calculation are just as flawed, and are being dangerously misused. Let us revisit Dr. Tom, the former CDC director.

Dr. Tom Frieden @DrTomFrieden

It's actually pretty simple math. If you get Covid and you've been vaccinated, you're about 100 times less likely to die.

Hyper-vigilant mass-vaccination advocates have been using the vaccine efficacy “from death” numbers in this Israeli study to claim that if you get COVID you have additional protection because of the vaccine. Some may quote the percentage (“it’s 96% effective at preventing death!”) so as to not be accused of inventing their own numbers. But if you have made it this far, you have performed the efficacy calculation three separate times now. Counting things and dividing things counted.

Nowhere, at any time, in any way, did we look at how many people in the vaccinated group got COVID and died. We (and the authors of the study) counted the number of deaths among the vaccinated, and divide by the number of days, not the number of infections.

And if we do actually look at and compare deaths among those actually infected we get a completely different story.

The efficacy numbers are misinformation

Every single additional efficacy number is redundant. All reported results downstream from infection are a consequence of reduced infections⁴. The efficacy rates would all be in the high 90s even if the vaccinated group were perfectly identical except for infection rate.

When “former Director of the CDC” Dr. Tom and others use the non-infection efficacy numbers to discuss the vaccines, they are, intentionally or not, misleading the public. It is something that should end immediately.

With the exception of infection rates, the efficacy numbers convey no useful information to citizens about their risks once they have been vaccinated. Instead, it may cause the vaccinated to place themselves and others at greater risk if they operate on this misinformation.

The efficacy numbers, other than infection, which are all downstream effects of infection, are being used and accepted as if they are additional, layered benefits at every stage of COVID infection.

This diagram is how they are being communicated and how they are being received by the clear majority of public figures and citizens. If you are vaccinated it is being communicated that the study showed that you are:

1. >90% less likely to get infected — true

2. >90% less likely to get symptoms if infected — false

3. >90% less likely to become hospitalized if you develop symptoms — false

4. >90% less likely to become severely hospitalized if you become hospitalized — false

5. >90% less likely to die you if become severely hospitalized — false

If the efficacy numbers were layered measures of additional protection then the observations would have been different in the Israeli study (All Ages)⁵.

With numbers this small, these differences may seem negligible. But when scaled up not just across populations but also across time, the impact of affecting people’s decision making when choosing what level of infection risk to assume, can dramatically alter the situation.

These are currently Acceptable Catastrophic Errors being made each and every day by nearly everyone. It is hard to fathom how much more wrong someone could be yet still maintain their credibility after this. But because these are acceptable catastrophic errors, if I am successful in clearing up confusion, those spreading the misinformation will simply move on to the next acceptable catastrophic error.

What the numbers really showed

When you actually take a look at who was infected and who died by age group, the idea of additional protection disappears for most.

When you are finally counting things and dividing things counted which matter, such as how many infected people went on to die in each group, no remnant of the 90% numbers remain. In the graph above⁶, there is no information available to suggest the death rate per infection is any different in the vaccinated group compared to the unvaccinated group. You can see why by revisiting the number infected and the number who died in each group.

36 of 84611 in the unvaccinated versus 0 in 1066 in the vaccinated group. 36 in 84611 is roughly 1 in 2350, but we only had 1066 infected in the vaccinated group. There is not enough information to claim the death rate per infection is higher or lower, and that uncertainty is indicated in the graph above. That is worlds away from the relative immortality communicated by the efficacy number 100%.

In fact, if there had been 24 deaths in the vaccinated group the efficacy reported would have been 3%! Because it was looking at rates over time, 24 deaths would have been the death rate over time similar to 36 in the unvaccinated group. But clearly, among those infected, 36 in 84611 is a far lower death rate than 24 in 1066!

The death rate for the ages 16-44 among infected would need to have been almost 53 times higher to report the vaccine was 3% effective at preventing death.

More graphs

Let’s look at some other situations.

For ages 45-64, there again isn’t enough evidence to conclude that the infection/case fatality rate is any different, but as the bluish dots indicate, the numbers we have actually lean towards a higher death rate among the vaccinated.

The silver lining though is that among those 65 and over, the observed death rate in the trial gives us decent reason to believe that maybe the vaccine can reduce the death rate for this group. Since they were the most vulnerable all along, this is a promising result.

Post hoc ergo propter hoc

If you were to notice that a number of people died after vaccination, and inferred that the vaccine must have caused the death, you would be committing what is known as the “post hoc ergo propter hoc” fallacy.

This is a simple enough fallacy to explain. Just because B happened after A does not mean B was caused by A. Causation requires more knowledge to deduce.

If you were to commit this fallacy communicating concern over vaccination you would be quickly told by pro-mass-vaccination advocates of this flaw in reasoning. However the same error is committed by the pro-mass-vaccination crowd without any internal pressure checking the reasoning.

We have great reason to assert a causal relationship between decreased infection rates and vaccination. The mRNA vaccines were designed to have a causal mechanism of interaction. Observing the intended effects of the causal mechanism on the macro level (fewer infections) is reasonable to attribute to the engineered mechanism on the micro level.

We have far less reason to assert causation downstream from preventing infection. The pro-vaccinate-everyone crowd has used their incorrect understanding of how the efficacy numbers were derived, and made post hoc ergo propter hoc arguments with them. Even when the data does not show any difference, a misreading and misunderstanding of efficacy numbers is automatically combined with post hoc ergo propter hoc reasoning, compounding errors.

These are all Acceptable Catastrophic Errors, unfortunately.

The dangers of efficacy misinformation

On any given day on social media, you can find someone who has been vaccinated saying they contracted COVID, and they just know that but for the vaccine they would have been in worse shape. These announcement get shared widely. It is wonderful that people have avoided some of the worst symptoms, but the data has not supported the observation that the vaccine offers significant downstream benefits for most people.

The repetition of this false idea, that there are amazing and significant additional benefits outside of mere reduced infection rates, will result in people taking increased risks.

The person who incorrectly assumes that “with the vaccine, even if I get infected I am less likely to die than I would be without the vaccine,” will be more likely to engage in behavior that risks infection. They may become the “it’s just a flu bro” of the vaccinated.

If the lack of observed layered protection were made more explicit then people may choose to still wear masks and social distance, to further reduce their exposure in addition to vaccination. They may choose to go to fewer social events. This in turn would reduce the infection rate, and reduce the probability a mutation that evades vaccination finds a host to practice on. (While writing this article, the White House changed its masking recommendation for the vaccinated)

Those who strongly believe everyone should get vaccinated may still choose to promote the efficacy numbers as additional layers of protection, because their goal is to encourage people to get vaccinated. Misleading use of numbers in advertising is a tale as old as time. But they are doing so at the cost of both their honesty and integrity, and sacrificing some of the actual benefits that vaccination provides, by giving people incorrect information to base their choices on.

Individual probabilities vs rates over time

Those who like to say “the numbers speak for themselves” are the least likely to understand what the numbers are saying. The studies that make epidemiological assessments are reporting to a different audience than the numbers reported are being presented to. To the extent that some studies may exist where these values are calculated conditionally, they are incompatible with the ones that have been computed unconditionally. This information is unlikely to be contained in the spreading of these numbers, as they are more often going to be used interchangeably.

The wrong audience

A hospital trying to plan resources and staffing cares about the estimated daily hospitalization and death rate given how many people are vaccinated. A policy maker deciding among options to fight coronavirus, cares what the daily rates are for various metrics after a policy is adopted.

An individual, for the most part, does not care about the daily rate in the population. They care what their own odds, or those of their loved ones, are.

It does not matter to an individual if the daily observed rate of infection is low across all people, if they themselves are at significantly higher risk. However, absent other measures that can guide an individual risk assessment, a report of daily infection rates if vaccinated compared to unvaccinated does give someone a useful measure at the level of making the choice to get vaccinated.

But once an individual becomes vaccinated, the daily rate of hospitalization, or the daily rate of death, is meaningless. Except in the extremes, such as a resources being exhausted, the daily rate of death has zero relationship to an individual’s chances of death once infected. The individual needs to know their probabilities if they ultimately become infected.

We’re being set up for disaster

At the beginning of this article I teased that the data show that death rate among the vaccinated and infected is higher than that of the unvaccinated and infected. This is true for death and also true for hospitalization.

There is no sleight of hand or trickery involved. These results are plain as day in the Israeli data if anyone had cared to look. It’s just the result of counting things and dividing things counted that people actually wanted to know about. Not only that, but these results are logical, and now easily predictable. They are also happening elsewhere and are causing confusion.

Why are the infection death rates higher?

The reason, hidden in plain sight, is that a large number people who were never going to die, are no longer getting infected.

By obsessively spreading the misinformation that people have additional benefits to protect against death if they become infected, in order to encourage everyone to get vaccinated, it becomes a unknown fact those most likely to die (6.6 in 100,000)⁷ are now most likely to get infected (2.5 in 100,000).

Contrast that with those who were least likely to die (0.04 in 100,000) are now those least likely to get infected (1.7 in 100,000).

And these results are now being reflected across the globe. In the UK, which is tracking variants, reports that among Delta variant cases, those vaccinated with two doses died at a rate of 0.0078 per infection and those unvaccinated died at a rate of 0.0014 per infection. The death per infection was 5.7 times higher among the vaccinated.

Without careful control and understanding, one might erroneously conclude the Delta variant is is more lethal if you’ve been vaccinated, the vaccine is losing its efficacy, the vaccination is making people weaker, or some combination. While any of those are possible outcomes in this environment, by not being aware of the infection death rate issue from the start, because one is busy spreading misinformation about extra levels of protection that the data do not support, one misses how to properly control for these effects and analyze new data as it comes in.

The death rate if infected was always going to be higher in the vaccinated groups if most of the vaccinated were those likely to die in the first place.

Those darn Acceptable Catastrophic Errors

Some people may simply shrug when this issue is pointed out to them. Some who understood at a casual level the that efficacy measures were rates over time, may simply not understand the big deal about using them to promote the vaccines. To some degree the vaccines change the probability someone who takes them will wind up dead from COVID. Is it really a big deal that the numbers are all double, triple, and quadruple counting decreased infections as additional benefits downstream? Is it really a problem if you tell a person they’ll be 100 times less likely to die if they get COVID and are vaccinated if that convinces them to get vaccinated?

Yes.

1. Initial mass surveillance study from Israel shows roughly 3 times higher, newest data from the UK shows 5.7 infection death rate among the vaccinated.

2. What does 95% COVID-19 vaccine efficacy really mean? - The Lancet, March 8, 2021

3. The authors mention they use a negative binomial regression in Stata to calculate the confidence intervals, and it is possible their point estimate also is a byproduct of this regression. Without access to files to re-run the regression, at this time it cannot be said if it is error, or the regression, that is the source of the difference.

4. This can be proved or demonstrated in many ways. But for those with familiar a bit of probability theory. If E_n is any chosen downstream event and E_1 is infection

Without significant effort to untangle downstream effects, the efficacy numbers other than infection will be repeating the infection efficacy.

5. First assume hospitalization/infection rate is the same as unvaccinated, then apply 98% protection on top. (5556/109876)*0.02 = 3.7

6. Beta posterior distribution with a Jeffreys prior. 95% Credible Interval

7. The evidence that exists in this data that among those 65 and older vaccinated of a decreased death rate around 50% cuts their rate to around 3.3 in 100,000. This is good, but still far higher than the other groups and enough to create the end result.