Week 11: bona fides and mala fides in peer review

Like Portia (and for the exact same reasons – my own group presented on the same topic in the morning Foundations class) peer review in the context of the debates over Open Access has been at the front of my mind of late. In her post, Portia brings up a very recent case, not unlike the Sokal affair: John Bohannon’s “sting” on Open Access journals in Science. I would like to use my blog entry to build on her commentary.

One interesting detail about Bohannon’s project is that out of the 304 Open Access journals he submitted his bogus article to, he knowingly included 122 journals listed at the time on Jeffery Beall’s list of predatory publishers (http://scholarlyoa.com/publishers/). Beall’s list has been criticized in its own right (generally for being too inclusive); but the fact remains that 40% of Bohannon’s submissions were to journals already known for, or at least suspected of unscrupulous, exploitative, & otherwise predatory practices. 82% of the journals on Beall’s list that completed some kind of review process accepted the paper – compared with 45% of journals listed in the Directory of Open Access Journals (and there is overlap between the two lists – I couldn’t find the numbers exclusively for journals not on Beall’s list). It seems to me that the real control group for this experiment were the Open Access publishers not on Beall’s list, and the numbers in Bohannon’s report are more effective at indicating the validity of Beall’s list than they are able to indicate anything about Open Access publishing in general.

Science magazine’s vested interest in promoting the traditional over Open Access publishing model may have contributed to the spin on Bohannon’s results (i.e. choice of title for the article). It’s difficult to say. In any case, the role of Science magazine in all this is rendered rather reflexive when one puts the flaws in Bohannon’s own “experiment” in the context of the journal’s own notorious misadventures in the peer-review process.

The question at the heart of Bohannon’s experimental design and Science‘s role in the publication of his results is whether Bohannon and Science magazine were truly acting in good faith. And here’s a thread that I’ve seen running through in the discourse on peer review, made most explicit in Stanley Fish’s NY Times piece on the Sokal Affair: trust. Both traditional scholarly publishing, and suggested alternative models depend to some extent on being able to trust that the various parties involved are acting in good faith.

Traditional models of peer review rest on the assumptions that authors are providing work they honestly believe in (including real data and a genuine attempt to accurately portray methodology and analysis), and that editors and peer reviewers will honestly, to the best of their knowledge and ability, determine whether an article is worth presenting to a wider audience, or whether it will waste the academic community’s time (and, with non-Open Access, money).

Alternative models of peer review put trust in slightly different areas, but being able to rely on everyone acting in good faith is no less important. The kind of open review described by Fitzpatrick depends on reviewers engaging with the process in good faith. For open peer review to work, those involved need to be both honest in their criticisms, and neither take advantage of knowing which criticisms come from whom (especially regarding young and otherwise vulnerable scholars) nor expect others to take such advantage.

I suspect that the publishing of advancements in highly specialized fields of knowledge will always depend, to some extent, on the assumption that most everyone involved is acting in good faith. As long as this is the case, peer review (no matter the business model and rights management practices of the journal involved) will always be vulnerable to theatrical stunts by the likes of Sokal and Bohannon.

Peer review, though clearly in need of some re-thinking, can only ever be one check point at the start of the long road to acceptance by the scholarly community. In many ways, peer review is just the beginning of that process. The fact is that the bulk of the burden lies (and in one way or another, will always lie) with the community at large to decide whether a bit of research is or is not bunk.

For Bohannon’s own summary and explanation, see: http://www.sciencemag.org/content/342/6154/60.full

For the most succinct criticism I’ve ever read of the “sting” operation, as well as an up-to-date list of other responses to Bohannon, see: http://svpow.com/2013/10/03/john-bohannons-peer-review-sting-against-science/

Cited:

Fitzpatrick, K. (2009). Chapter 1: Peer review. In Planned Obsolescence: Publishing, Technology, and the Future of the Academy.