Populations and Samples: The Bakersfield COVID-19 Data Controversy

In April 2020, Drs. Dan Erickson and Artin Massihi, who run Accelerated Urgent Care (a private medical clinic in Bakersfield, Kern County, California) posted a video in which they argued that the death rate and prevalence of Covid-19 were comparable to seasonal flu. It was already so widespread, they claimed, that "herd immunity" had developed and nobody needed to fear the new coronavirus. Their video quickly garnered many views and they were featured on Fox News. Almost as quickly, questions were raised about their data.

According to Dr Erickson, Kern County had conducted 5,213 Covid-19 tests which revealed 340 cases.

Question 1: What can we conclude from these figures?

Answer: We can conclude that 340/5213 (as a decimal, about 0.065) or about 6.5% of the individuals tested had the novel coronavirus.

Question 2: What other information would help us understand the practical significance of this result?

NOTE: The 2020 population of Kern County was about 912,316 people.

Question 3: In the original video, Dr. Erickson says, "That’s 6.5% of the population.” Is that a correct statement? Why or why not?

Answer: "Population" is a technical term in Statistics. It means the entire collection of individuals of interest in the study. We are most likely interested in "all Kern County residents" as our population.

It was NOT true that 6.5% of Kern County residents (all the individuals of interest) tested positive for Covid-19. It was 6.5% of the people who were tested.

In epidemiological terms, 6.5% is the "test-positive rate." It is not the "prevalence" (as Dr. Erickson wrongly stated).

Then Erickson stated that in California as a whole, there had been 33,865 cases out of 285,900 tests.

Question 4: Assuming his figures were accurate, what is the correct interpretation?

Answer: About 11.8% of the people tested for Covid-19 in California were found to have the novel coronavirus. The test-positive rate was about 11.8%.

Question 5: Dr. Erickson said that it meant that "12% of Californians are infected." What is wrong with his statement?

Question 6: What would have to be true in order for a "test-positive rate" to accurately represent the prevalence of the disease in the population?

Answer: The people tested were a sub-set of the entire population. The statistical term is that the persons tested were a SAMPLE of the POPULATION. The test-positive rate would only represent the prevalence of the disease IF the sample was representative (typical) of the general population.

Question 7: Do you think it likely that the sample of persons tested for Covid-19 in Kern County (or in all California) in April 2020 was a representative sample of the whole population? Why or why not?

Answer: No! It was not easy to get a Covid-19 test in April 2020 anywhere in the USA. Only people who were showing disease symptoms or who knew they had probably been exposed were able to get the tests. Those tested were much more likely to be sick than the general population.

Question 8: If the sample of people tested was expanded so that people without symptoms or known exposure got tested, would you expect the positivity rate to go up, down, or stay the same? Why?

Answer: It would go down. Imagine that Kern County tested another 5,213 people but, because this sample included more healthy people, only 100 more cases were found. Then the positivity rate would be 440/10,426 or about 4.2%. (Dr. Erickson got this wrong. He said that “the more you test, the prevalence estimate goes up.”)

Question 9: You have probably heard it said that the US has so many COVID-19 cases because there is so much testing going on. Apart from the fact that the amount of testing is NOT very large relative to the US population, what is wrong with this argument?

Answer: Nobody gets Covid-19 from being tested. As more people are tested, more of the already-existing cases are found. If there were zero testing, we would find zero cases but there still would be hundreds of thousands of them.

Question 10: Can you think of a way to test a truly representative sample of people in Kern County (or in California)? What might make that difficult?

Answer: The "gold standard" for obtaining a representative sample is a SIMPLE RANDOM SAMPLE. In theory, for example, one could use a random-number generator to generate a list of social security numbers, and test Californians with a SSN on that list. The practical difficulties, however, would be overwhelming.

Other sampling techniques (such as cluster sampling and stratified samples) are used when a simple random sample is not practical. However, it is never appropriate to make an inference about a population (such as prevalence) from a sample that is clearly unrepresentative in an important way, as the Bakersfield doctors did.

If you would like to discuss this article with me or your classmates, please go here: Discussion: Data questions about Covid-19