How not to critique: a case study

The original title for this piece was “How not to critique in bioethics,” but Kelly pointed out that this episode of TWiV is a case study in how not to go about critiquing anything.

Last Monday I was drawn into a conversation/angry rant about an article by Lynn C. Klotz and Edward J. Sylvester, that appeared in the Bulletin of the Atomic Scientists…in 2012. After briefly forgetting one of the cardinal rules of the internet—check the date stamp— I realized the error of my ways, and started to inquire with my fellow ranters, in particular Matt Freiman, about why a 2012 article suddenly had virologists up in arms.

Turns out that the Bulletin article was cited by a study on dual-use authored by Marc Lipsitch and Alison P. Galvani; a study that was the subject of a recent post of mine . The Bulletin article draws from a working paper where the provide an estimate for the number of laboratory accidents involving dangerous pathogens we should expect as a function of hours spent in the laboratory. Lipsitch and Galvani use this figure in their analysis of potential pandemic pathogens (PPPs).

Freiman joined Vincent Racaniello, Dickson Despommier, Alan Dove, and Kathy Spindler on This Week in Virology (TWiV) on June 1 to talk about (among other things) Lipsitch and Galvani’s research. What followed is a case study in how not to critique a paper; the hosts served up a platter of incorrect statements, bad reasoning, and some all-out personal attacks.

I’d started writing a blow-by-blow account of the entire segment, but that quickly mushroomed into 5,000-odd words. There is simply too much to talk about—all of it bad. So there’s a draft of a paper on the importance of good science communication on my desk now, that I’ll submit to a journal in the near future.Instead, I’m going to pick up just one particular aspect of the segment that I feel demonstrates the character of TWiV’s critique.

“It’s a bad opinion; that’s my view.”

Despommier, at 58:30 of the podcast, takes issue with this sentence in the PLoS Medicine paper:

The H1N1 influenza strain responsible for significant morbidity and mortality around the world from 1977 to 2009 is thought to have originated from a laboratory accident.

The problem, according to Despommier, is that “thought to have originated” apparently sounds so vague as to be meaningless. This leads to a rousing pile-on conversation in which Despommier claims that he could have just easily claimed that the 1977 flu came from Middle Eastern Respiratory Syndrome because “he thought it;” he also claims that on the basis of this sentence alone he’d have rejected the article from publication. Finally, he dismisses the citation given in the article as unreliable because it is a review article,[1] and “you can say anything in a review article.”

At the same time, Dove notes that “when you’re on the editorial board of the journal you can avoid [having your paper rejected].” The implication here is that Lipsitch, as a member of the editorial board of PLoS Medicine, must have used that position to get his article to print despite the alleged inaccuracy that has Despommier so riled up. Racaniello notes that “[statements like this are] often done in this opinion–” his full meaning is interrupted by Despommier. It’s a common theme throughout the podcast, though, that Lipsitch and Galvani’s article is mere “opinion,” and thus invalid.

Facts first

If he’d done his homework, Despommier would have noted that the review article cited by Lipsitch and Galvani doesn’t mention a lab. What it does say is:

There is no evidence for integration of influenza genetic material into the host genome, leaving the most likely explanation that in 1977 the H1N1 virus was reintroduced to humans from a frozen source.[2]

So Lipsitch and Galvani do make an apparent leap from “frozen source” to “lab freezer.” Despommier doesn’t pick that up. If he had, however, it would have given us pause about whether or not is a valid move to jump from “frozen source” to “laboratory freezer.”

Not a long pause, however; there are other sources that argue that the 1977 strain is likely to have been a laboratory.[3] The other alternative—that the virus survived in Siberian lake ice—was put forward in a 2006 paper (note, after the publication of the review article used by Lipsitch and Galvani), but that paper was found to be methodologically flawed.[4] Laboratory release remains the most plausible answer to date.

The belief that the 1977 flu originated from frozen laboratory sources is widely held. Even Racaniello—at least, in 2009—holds this view. Racaniello argued that of multiple theories about the origin of the 1977 virus, “only one was compelling”:

…it is possible that the 1950 H1N1 influenza virus was truly frozen in nature or elsewhere and that such a strain was only recently introduced into man.

The suggestion is clear: the virus was frozen in a laboratory freezer since 1950, and was released, either by intent or accident, in 1977. This possibility has been denied by Chinese and Russian scientists, but remains to this day the only scientifically plausible explanation.

So no, there is no smoking gun that confirms, with absolutely unwavering certainty, that the 1977 flu emerged from a lab. But there is evidence: this is far from an “opinion,” and is far from simply making up a story for the sake of an argument. Lipsitch and Galvani were right to write “…it is thought,” because a plausible answer doesn’t make for unshakeable proof—but their claim stands on the existing literature.

Science and policy

The idea that Lipsitch and Galvani’s piece is somehow merely “opinion” is a hallmark of the discussion in TWiV. Never mind that the piece was an externally peer-reviewed, noncommissioned piece of work.[5] As far as TWiV is concerned, it seems that if it isn’t Science, it doesn’t count. Everything else is mere opinion.

But that isn’t how ethics, or policy, works. In ethics we construct arguments, argue about the interpretation of facts and values, and use that to guide action. With rare exception, few believe that we can draw conclusions about what we ought to do straight from an experiment.

In policy, we have to set regulations and guidelines with the information at hand—a policy that waits for unshakeable proof is a policy that never makes it to committee. Is there some question about the true nature of the 1977 flu, or the risks of outbreaks resulting from BSL–3 laboratory safety? You bet there is. We should continue to do research on these issues. We also have to make a decision, and the level of certainty the TWiV hosts seem to desire isn’t plausible.

Authority and Responsibility

This podcast was irresponsible. The hosts, in their haste to pan Lipsitch and Galvani’s work, overstated their case and then some. Dove also accused Lipsitch of research misconduct. I’m not sure what the rest of the editors at PLoS Medicine think of the claim—passive aggressive as it was—that one of their colleagues may have corrupted the review process, but I’d love to find out.

The podcast is also deeply unethical, because of the power in the platform. Racaniello, in 2010, wrote:

Who listens to TWiV? Five to ten thousand people download each episode, including high school, college, and graduate students, medical students, post-docs, professors in many fields, information technology professionals, health care physicians, nurses, emergency medicine technicians, and nonprofessionals: sanitation workers, painters, and laborers from all over the world.[6]

What that number looks like in 2014, I have no idea. I do know, however, that a 5,000–10,000 person listenership, from a decorated virologist and his equally prestigious colleagues, is a pretty decent haul. That doesn’t include, mind you, the people who read Racaniello’s blog, articles, or textbook; who listen to the other podcasts in the TWiV family, or follow the other hosts in other fora.

These people have authority, by virtue of their positions, affiliations, exposure, and followings. The hosts of TWiV have failed to discharge their authority with any kind of responsibility.[7] I know the TWiV format is designed to be “informal,” but there’s a marked difference between being informal, and being unprofessional.

Scientists should—must—be part of conversation about dual-use, as with other important ethical and scientific issues. Nothing here is intended to suggest otherwise. Scientists do, however, have to exercise their speech and conduct responsibly. This should be an example of what not to do.

Final Notes

I want to finish with a comment on two acts that don’t feature in Despommier’s comments and what followed, but are absolutely vital to note. The first is that during the podcast, the paper by Lipsitch and Galvani is frequently referred to as “his” paper. Not “their” paper. Apparently recognizing the second—female—author isn’t a priority for the hosts or guests.

Also, Dove and others have used Do Not Link (“link without improving ”their“ search engine position”) on the TWiV website for both the paper by Lipsitch and Galvani, and supporting materials. So not only do the hosts and guests of the show feel that the paper without merit; they believe that to the point that they’d deny the authors—and the journal—traffic. Personally, I think that’s obscenely petty, but I’ll leave that for a later post.

Science needs critique to function. Critique can be heated—justifiably so. But it also needs to be accurate. This podcast is a textbook example of how not to mount a critique.

---

1. Webster, Robert G, William J Bean, Owen T Gorman, Thomas M Chambers, and Yoshihiro Kawaoka. 1992. “Evolution and Ecology of Influenza A Viruses” Microbiological Reviews 56 (1). Am Soc Microbiol: 152–79.  ↩
2. ibid., p.171.  ↩
3. Ennis, Francis A. 1978. “Influenza a Viruses: Shaking Out Our Shibboleths.” Nature 274 (5669): 309–10. doi:10.1038/274309b0; Nakajima, Katsuhisa, Ulrich Desselberger, and Peter Palese. 1978. “Recent Human Influenza a (H1N1) Viruses Are Closely Related Genetically to Strains Isolated in 1950.” Nature 274 (5669): 334–39. doi:10.1038/274334a0; Wertheim, Joel O. 2010. “The Re-Emergence of H1N1 Influenza Virus in 1977: a Cautionary Tale for Estimating Divergence Times Using Biologically Unrealistic Sampling Dates.” PLoS One 5 (6). Public Library of Science: e11184. doi:10.1371/journal.pone.0011184; Zimmer, Shanta M, and Donald S Burke. 2009. “Historical Perspective — Emergence of Influenza a (H1N1) Viruses.” New England Journal of Medicine 361 (3): 279–85. doi:10.1056/NEJMra0904322.  ↩
4. Worobey, M. 2008. “Phylogenetic Evidence Against Evolutionary Stasis and Natural Abiotic Reservoirs of Influenza a Virus.” Journal of Virology 82 (7): 3769–74. doi:10.1128/JVI.02207–07; Zhang, G, D Shoham, D Gilichinsky, and S Davydov. 2007. “Erratum: Evidence of Influenza a Virus RNA in Siberian Lake Ice.” Journal of Virology 81 (5): 2538; Zhang, G, D Shoham, D Gilichinsky, S Davydov, J D Castello, and S O Rogers. 2006. “Evidence of Influenza a Virus RNA in Siberian Lake Ice.” Journal of Virology 80 (24): 12229–35. doi:10.1128/JVI.00986–06.  ↩
5. I’m aware that peer review is not sufficient to make a work reliable, but absent evidence that the review process was somehow corrupt or deficient, it’s a far cry from mere opinion.
6. Racaniello, Vincent R. 2010. “Social Media and Microbiology Education.” PLoS Pathogens 6 (10). Public Library of Science: e1001095.  ↩
7. Evans, Nicholas G. 2010. “Speak No Evil: Scientists, Responsibility, and the Public Understanding of Science.” NanoEthics 4 (3): 215–20. doi:10.1007/s11569–010–0101-z.  ↩