The short version: Vaccine efficacy is not measured by how many people have to get a vaccine to prevent a case of COVID-19 vaccine. These models don’t take into account that vaccines exert protective effects on the community and so the more people get a vaccine, the fewer people have to get it to prevent a single case of a vaccine, which is the opposite of what these models would imply. Also, BMJ Rapid Responses are not peer-reviewed papers, so be wary of people trying to misrepresent them as such.

Some individuals have latched onto a Rapid Response from the BMJ that suggests that one has to vaccinate 256 individuals for Pfizer’s vaccine to prevent one case of COVID-19. This is a not an accurate way to interpret vaccine efficacy.

Firstly, before we start dissecting all the ways in which this is wrong, we need to talk about the Rapid Response feature of the BMJ. Here is what the BMJ says about their rapid responses (emphasis mine):

Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully [sic] misrepresented as published articles.

The rapid response feature is like a youtube comment. No one need have any evidence, credibility in the field to make these, and because the BMJ’s name appears on the page, it gives a semblance of credibility to the work, and these are frequently misrepresented as being peer-reviewed articles published in the journals. Anyway, let’s discuss the substance of the matter.

Cunningham is attempting to use a concept called the number needed to treat (NNT). This is the number of individuals who need to get a particular intervention to prevent a single case of a bad outcome. The number needed to treat is taken by taking the reciprocal of the absolute risk reduction (ARR) for an intervention. For example, if the baseline risk of a bad outcome is 40%, and the treatment reduces that to 20%, the absolute risk reduction is 40-20 = 20%. The number needed to treat is therefore 1/0.20 = 5. Thus for every 5 people who get the intervention, one will not get the bad outcome in question. Here is Pfizer’s phase 3 data:

• 170 cases of COVID-19

• 162 in placebo group

• 8 in vaccinated group

• 10 severe cases of COVID-19; 9 in placebo, 1 in vaccinated group.

• ~43,000 subjects, with 1:1 randomization, so ~21,500 people per group

As I discussed before, these are normal results for a vaccine (a VERY effective vaccine) trial. The major reason you need to recruit so many people in a phase 3 study of vaccine efficacy is simply that the event you are interested in- someone getting the disease (that you hope the vaccine prevents)- is rare. The efficacy for a vaccine at preventing COVID-19 is given by:

This quantity is equivalent to the relative risk reduction (RRR) for the vaccine at the given intervention. However, NNT is based on the absolute risk reduction (ARR), and this changes the math quite a bit. ARR is given by:

In other words, ARR does not adjust for the control event rate.

We will forgive Cunningham his lack of clairvoyance that the NNT (or number needed to vaccinate, NNV, which is calculated in the same way but applied specifically to vaccines measures how well they prevent an outcome) would change between the interim and final efficacy analysis. These data would suggest that you would need 140 people to be vaccinated against COVID-19 to be able to prevent a single case of COVID-19 with the Pfizer vaccine, which paints a far more grim picture of the value of this vaccine. Cunningham’s argument however, is at best, plainly ignorant, and at worst, disinformation. Here is why.

First of all, the trial of ~42,000 people was stopped when there were 170 cases. If hypothetically all of the cases occurred in the unvaccinated group, the NNT would be 124, which is the result we would expect if this vaccine had 100% efficacy at preventing COVID-19. I raise that point for 3 reasons reasons:

1. There isn’t all that much difference between needing to treat 140 people vs. needing to treat 124 people to see (or in this case, prevent) an outcome. Saying this vaccine doesn’t work when it works pretty close to as well as is literally theoretically possible is an unreasonable argument. The next step in the logical sequence would be that a vaccine isn’t necessary at all, but a vaccine has been the exit strategy for the pandemic since the start. There’s a very good reason for that: vaccines work.

2. ARR and NNT are a function of the incidence of the phenomenon in question. By design, these trials stop at a relatively small number of cases, and in any given vaccine trial this will be rare, and if allowed to go for longer ARR would necessarily increase and NNT would drop.

3. This is not how vaccine efficacy is computed.

At any given moment, the incidence of any infectious disease is going to be relatively low. The problem is… it’s infectious. And as COVID-19 has demonstrated repeatedly, until there is some appreciable herd protection effect from people becoming immune to it (we are nowhere close), the growth is exponential. These metrics cannot take into account the indirect effects of vaccination. As people are immunized, vaccines exert a protective effect on the people around the vaccinee, as the vaccinee becomes less effective at passing the infection in question than at baseline. In more formal terms, we would say that the force of infection declines as a population gains immunity. This alone invalidates ARR as a measure as ARR does not consider external factors such as the proportion of the population that is immunized. In fact, this would drive down ARR because the disease in question becomes rarer. The irony of this can be seen with influenza vaccines. While certainly better than nothing, they leave much to be desired in terms of preventing influenza, and have previously been measured as having an NNT of 5. It clearly makes no sense that influenza vaccines, which achieve an RRR of about 40-60% most years, should be considered more efficacious at reducing the prevalence of influenza, than a COVID-19 vaccine that has an RRR of 95.1%.

We can take an even more extreme case. There are no polio cases in the US and have not been any since 1993. If you apply Cunningham’s methodology, the NNT is infinite and there is no value to getting a polio vaccine in the US. Except we know this isn’t true because polio is still endemic in some parts of the world and if vaccination were to abruptly stop, it would make a resurgence (though there is good news in that polio is eradicable). Furthermore, the reason that there is no polio in the US is because of the polio vaccine.

Biostatistician Frank Harrell also has some well-reasoned qualms with NNT as a concept. The argument is that it’s easier to interpret than ARR for clinicians. However, ARR as a quantity totally loses its frame of reference. For instance, if the risk drops from 0.02 to 0.01, NNT = 100. But it’s a very different thing from when ARR drops from 0.93 to 0.92. Another issue is that ARR is based on populations rather than individuals, but clinicians work with individuals to make decisions. Extrapolating to them the individual is therefore problematic as the individual in question may not represent the “average” from the group the ARR is based on. This “average” individual may in fact not correspond meaningfully to any real person. Additionally, on the rare instances where a confidence interval comes with an ARR, if it contains 0, the confidence interval for the NNT will contain infinity, which is not meaningful (you would need to treat between x and infinity people to prevent bad outcome y).

In addition, NNT is redundant. It contributes no additional information compared with ARR.

Anyway, the simplest answer to the question in the title is: the more people get vaccinated, the fewer people will additionally need to be vaccinated to prevent a case of COVID-19.

Several ancillary points are made by Cunningham that are erroneous but worth addressing in brief.

Firstly, while ADE had been observed in preclinical models for some coronavirus vaccines, these do not predict the potential of ADE in humans. Even in circumstances in which a prior infection exacerbates disease, the explanation is rarely an antibody-mediated process. The nature of ADE relies in part on the presence of antibodies which bind, but fail to neutralize viral particles. The antibodies elicited by vaccines containing the RBD of the spike protein necessarily will be neutralizing, because they prevent association of the RBD with viral receptors. The key point is given by Arvin et al:

The administration of passive antibodies could also reveal whether antibodies predispose to ADE of disease…
The evidence that COVID-19 does not worsen after treatment with plasma from convalescent patients has been substantially reinforced by a study of 20,000 patients who were severely ill with the disease, showing an adverse event incidence of 1–3% [81]. If further substantiated, these findings will markedly diminish the concern that clinically relevant amplification of infection, release of immunopathogenic cytokines or immune-complex deposition in the presence of a high viral load is mediated by SARS-CoV-2 antibody-dependent mechanisms [82,83].

Squalene is a precursor to cholesterol (meaning everyone has it in their body because everyone makes cholesterol), and it is present in the MF59 adjuvant which is used in the adjuvanted flu vaccine given to individuals older than 65. Despite his comments, it is decidedly not the cause of Gulf War syndrome. As White et al describe

Across all studies and populations and since the earliest findings appeared linking self-reported exposures to diagnosis of GWI, two types of theater-related exposures have been consistently identified as risk factors for the disorder: exposures to pesticides and PB [pyridostigmine bromide] use.

PB is an inhibitor of the acetylcholinesterase enzyme, which breaks down the compound acetylcholine that tells muscles to contract. It was given prophylactically in case of nerve agent use.

The idea that COVID-19 vaccine trials cannot make a determination about whether or not the vaccines in question prevent severe COVID-19 is simply incorrect. This may not have been the primary endpoint for the trials of most vaccines (which was to prevent COVID-19 of any severity), but this was absolutely monitored (or else we would not have a breakdown from either Pfizer or Moderna on who had severe disease). Furthermore, as I described in greater detail here, preventing severe disease is generally easier than preventing mild disease for a vaccine.

The notion that influenza vaccines worsen mortality in the elderly is absolutely untrue, and is furthermore not the conclusion of the Anderson et al study Cunningham cites. There are multiple studies that indicate that influenza vaccines have important protective effects on severe disease outcomes.

Acknowledgments

I extend my sincerest gratitude to Dr. Rene Najera, who checked this piece for accuracy and made sure my reasoning did not become circular, even when it was very tempting. I must also thank Victoria Crabb MS, BSN for checking my epidemiology and her unwavering support.

References

1. A vaccine is the one true global exit strategy from this pandemic, but timeline is frustratingly long. Khn.org. 2020 Apr 7 [accessed 2020 Dec 3]. https://khn.org/morning-breakout/a-vaccine-is-the-one-true-global-exit-strategy-from-this-pandemic-but-timeline-is-frustratingly-long/

2. Agarwal R. Confidence intervals explained simply for data scientists. Towards Data Science. 2019 Dec 23 [accessed 2020 Dec 3]. https://towardsdatascience.com/confidence-intervals-explained-simply-for-data-scientists-8354a6e2266b?gi=56a16e534896

3. Ameratunga R, Gillis D, Gold M, Linneberg A, Elwood JM. Evidence refuting the existence of autoimmune/autoinflammatory syndrome induced by adjuvants (ASIA). The journal of allergy and clinical immunology in practice. 2017;5(6):1551-1555.e1.

4. Anderson ML, Dobkin C, Gorry D. The effect of influenza vaccination for the elderly on hospitalization and mortality: An observational study with a regression discontinuity design. Annals of internal medicine. 2020;172(7):445–452.

5. Arvin AM, Fink K, Schmid MA, Cathcart A, Spreafico R, Havenar-Daughton C, Lanzavecchia A, Corti D, Virgin HW. A perspective on potential antibody-dependent enhancement of SARS-CoV-2. Nature. 2020;584(7821):353–363.

6. Benefits of Influenza Vaccination: Selected publications. Cdc.gov. 2019 Apr 19 [accessed 2020 Dec 3]. https://www.cdc.gov/flu/about/qa/benefit-publications.htm

7. Berg JM, Tymoczko JL, Stryer L. Cholesterol is synthesized from acetyl coenzyme A in three stages. New York, NY: W.H. Freeman; 2002 [accessed 2020 Dec 3].

8. CDC. Polio Elimination in the U.S. Cdc.gov. 2020 Jul 18 [accessed 2020 Dec 3]. https://www.cdc.gov/polio/what-is-polio/polio-us.html

9. Covid-19 vaccine candidate is unimpressive: NNTV is around 256. 2020 [accessed 2020 Dec 3]. https://www.bmj.com/content/371/bmj.m4347/rr-4

10. Doeschl-Wilson A. Lecture 5: Deterministic compartmental epidemiological models in homogeneous populations. Edu.au. [accessed 2020 Dec 3]. https://jvanderw.une.edu.au/L5_ModellingEpidemics1.pdf

11. Endemic Countries. Polioeradication.org. [accessed 2020 Dec 3]. https://polioeradication.org/where-we-work/polio-endemic-countries/

12. Ko E-J, Kang S-M. Immunology and efficacy of MF59-adjuvanted vaccines. Human vaccines & immunotherapeutics. 2018;14(12):3041–3045.

13. Pfizer and BioNTech conclude phase 3 study of COVID-19 vaccine candidate, meeting all primary efficacy endpoints. Pfizer.com. [accessed 2020 Dec 3]. https://www.pfizer.com/news/press-release/press-release-detail/pfizer-and-biontech-conclude-phase-3-study-covid-19-vaccine

14. Problems with NNT. Datamethods.org. 2018 Jul 24 [accessed 2020 Dec 3]. https://discourse.datamethods.org/t/problems-with-nnt/195

15. Recent rapid responses. Bmj.com. [accessed 2020 Dec 3]. https://www.bmj.com/rapid-responses

16. Tuite AR, Fisman DN. Number-needed-to-vaccinate calculations: fallacies associated with exclusion of transmission. Vaccine. 2013;31(6):973–978.

17. Vanderslott S, Dadonaite B, Roser M. Vaccination. Our World in Data. 2013 [accessed 2020 Dec 3]. https://ourworldindata.org/vaccination

18. Walker G, The NNT Group. Vaccines for preventing influenza in healthy individuals – TheNNT. Thennt.com. [accessed 2020 Dec 3]. https://www.thennt.com/nnt/vaccines-preventing-influenza-healthy-individuals/

19. White RF, Steele L, O’Callaghan JP, Sullivan K, Binns JH, Golomb BA, Bloom FE, Bunker JA, Crawford F, Graves JC, et al. Recent research on Gulf War illness and other health problems in veterans of the 1991 Gulf War: Effects of toxicant exposures during deployment. Cortex; a journal devoted to the study of the nervous system and behavior. 2016;74:449–475.