Each year the CDC publishes an estimate of the effectiveness of the flu vaccine in the previous flu season. Recently, the NIH director criticized the test-negative design from which the estimates are derived. He was right. The basic premise of the design is a two-edged sword: on the one hand, restricting the sample to people who sought medical care might reduce confounding by healthcare-seeking behavior; on the other hand, that restriction might add another type of bias — colliding bias — which is not as widely appreciated. The net bias remains unknown.
This, however, is not the only shortcoming of test-negative case-control studies of the flu vaccine. In this post, I will expose the shaky results of a large study of the flu vaccine in 2022–2023, when the vaccine was well-matched to the dominant strain. The study was based on the VISION Vaccine Effectiveness Network, one of several networks that collaborate with the CDC. Below are the published results.
Given a changing risk of infection, a valid comparison of the vaccinated and the unvaccinated requires similar distributions of the two populations over time. This is not the case because vaccination is associated with calendar time (rollout).
[The authors show the vaccination status at the time of seeking care, but most people got vaccinated by the end of December, and the percentage of vaccinated people stabilized in January at about 45% of the encounters.]
As shown below (Table), the share of the vaccinated population in October through December (47%), a period of high background risk, was lower than the comparable share in the unvaccinated population (61%). Of course, the complementary shares in January through March, a period of low risk, were reversed: 53% versus 39%.
