All of that might be moot if the testing devices used in both surveys turn out to be flawed. The results come amid widespread concerns about the accuracy of blood antibody tests—especially the rapid lateral flow tests like the ones used in this study. The Stanford preprint referred to a test from Premier Biotech, based in Minneapolis, but that company is only a distributor. The firm that makes the test, Hangzhou Biotest Biotech, was previously identified by NBC as among those recently banned from exporting Covid-19 tests because its product hasn’t been vetted by China’s equivalent of the FDA. A representative for Premier Biotech confirmed to WIRED that the same test was used by the Stanford and USC researchers. (On Monday, a USC spokesperson emailed WIRED a statement from Neeraj Sood, the lead researcher, acknowledging the test’s origins and noting they were exported legally, prior to the ban.)
At Stanford, the researchers performed their own validation of the tests and found only false negatives, not false positives. But, as a chorus of statisticians have noted (including one who concluded his analysis by demanding an apology from the researchers for wasting everyone’s time), even a low false positive rate would wipe out the significance of their results. Representatives for Stanford declined to make the researchers available for follow-up questions, including whether they planned to retest the positive individuals.
Reached by email, Sood wrote that his USC team did not plan to confirm the positive results with additional blood tests, but had offered genetic swab tests to those who had tested positive and might have current infections. Speaking at the press conference Monday, Sood acknowledged the results were “preliminary,” but said they fell in line with his expectations about the scale of untested infections.
The Stanford researchers said at their press conference on Friday that they’ve accounted for the potential biases in who volunteered and that they stood by their validation methods. They also pointed out that outcomes might be different in Santa Clara County than in overwhelmed New York City. “It seems to me that when the hospitals are overrun, that can make this epidemic a lot worse,” Bendavid said. The researchers do not intend to replicate the tests in Santa Clara, though they note others are conducting serosurveys in the Bay Area. In Los Angeles, they plan to do another round of tests in coming weeks.
That, at least, is one point everyone in this debate agrees on: More serosurveys are a good thing, and will ideally involve properly validated tests and representative populations. “We should be rolling out these tests to everybody,” said Martin Hibberd, an infectious disease researcher at the London School of Hygiene and Tropical Medicine.
While Hibberd called the methods used in the Santa Clara study “dubious,” he also said the group’s conclusions do likely point in the right direction—that the fatality rate will come down as more undetected cases are uncovered. That’s true of any disease outbreak. Other initial serosurvey results trickling in over recent weeks, from places like Denmark, Germany, the Netherlands, and the UK, as well as China, seem to back that up. While the rates of seroprevalence vary—from 1 percent among Scottish blood donors tested in mid-March to 15 percent in one hard-hit German town—they all generally point to lots of untested people. But it’s still a matter of how many. “It’s been guesswork really,” Hibberd added. “Just to see the first bit of data is quite exciting.”
Even assuming solid tests and methods, this first round of serosurvey results only gives us a snapshot of the disease’s progress in each place. That means uncertainties abound, Hibberd noted. Antibodies take time to develop, and people also take time to recover—or die—from the disease. We’re still at a relatively early stage of data collection, when the actual number of deaths due to Covid-19 remains uncertain. New York, for example, recently revised its death toll upward to include “likely cases” of Covid-19.
A better use for these very early results is the simple one: a rough estimate of much further we have to go in this pandemic. One hope among researchers is that we’ll get to a point of herd immunity, when enough people have antibodies to slow or stop the disease’s transmission. (That’s still just a hope, given uncertainties about whether antibodies correspond to actual immunity.) But so far, all the serosurveys show results in the single digits, which means the virus remains dangerous to most of the population.