Peer-Review, or: How I Learned to Stop Worrying and Accept the Flaws

When reviewing the qualities of a good peer-review, information during lecture was primarily drawn upon Michael Tyworth’s Blog post ‘How to Conduct a Peer Review’. As was mentioned, the first quality in a good review is to take the ‘I-want-my-paper-to-get-published’ approach straight to the reviewer. What I actually found most useful is when Prof. Galey made a thought-provoking critique of this point, stating that as researchers our mission is not to get published, but rather to ensure that our article enriches readership through its publication. We should want the journal to succeed first and foremost as a reliable source of knowledge advancement and enhancement, and not for the selfish, self-rectifying reasons that often lead authors to praise their work and remain ignorant of its flaws. My analogy here is that publishing an article requires the same care and thought as erecting a monument or statue, for who would agree to subject external spectators and consumers of knowledge to such an impermeable object without considering its validity, meaning, purpose and contribution. For this reason, it is important for the peer-reviewer to not only play the role of critic, but also be a coach who crafts advice in an accessible and useful way that motivates the author to improve the article.

With Prof. Galey’s critique style, a ‘peer-review of peer-review methods’, I’d like to bring to your attention an additional critique of review questioning methods. As mentioned in lecture, in the reviewing process it is crucial to ask several questions involving the research methods employed, the source of data (or how the data was collected and analyzed), whether the research design has reliability and whether the data has internal and external validity. The last point here is quite problematic and doesn’t go without mentioning that, for obvious reasons, there is no guaranteed way of regulating whether experimental research data itself is entirely valid. In this sense, I agree with YAAWESOMESAUCE’s post (a.k.a. Brooke Windsor) in that peer-review in scientific, experimental research fields requires constant review and validation beyond publication. William Y. Arms makes this fact clear in his 2002 article which I’ve linked below. He brings in the example of the Journal of the ACM, a highly validated journal of theoretical computer science. Arms states of a particular article he wrote for the journal that only twenty years later did he discover that the particular data-set he used from a prior research study was in fact ‘fraudulent’. The problem here is that one cannot blame the peer-reviewers, for as he states “[they] had no way of knowing this fact. The hypothesis in the paper has been confirmed by other experiments, but the erroneous paper — in a respectable, peer-reviewed journal — can still be found on library shelves.” This illuminates the bigger problem that one cannot simply answer whether data has internal and external validity, for a peer-reviewer must repeat the experiment in order to know how to answer this truly (a time consuming, costly, and nearly impossible feat). Unfortunately, this leaves erroneous information in so-called reliable, scholarly journals subject to consumption by the unsuspecting scholar.

In this case, Arms asserts that peer-review is “little more than a comment on whether the research appears to be well done”. What do you think? Would you agree with Arms’ assessment here?  Or should we all just stop worrying and accept the flaws of peer-review, for, truly, how else would we ever contribute to scholarship and bring new knowledge out to the world?

Olivia Wisniewski

Arms, William Y. (2002). “What are the alternatives to peer review? Quality Control in scholarly publishing on the web.” The Journal of Electronic Publishing, 8 (1). DOI: 10.3998/3336451.0008.103. Retrieved from: http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0008.103

Advertisements

2 Comments

Filed under Blogging Question

2 responses to “Peer-Review, or: How I Learned to Stop Worrying and Accept the Flaws

  1. Hey Olivia,

    I think the Arms statement you highlighted, about peer review speaking to research being well done and nothing more, shows his oversight that it does help to define the quality of the research. Allowing research to be reviewed acts as a buffer to ensure that the research was conducted in a thorough and appropriate way. Having other experts review the information before it become public knowledge makes sure that the information does hold a measure of accuracy. Re-testing that information helps to fortify that accuracy or provide information as to why it is incorrect. We learn from mistakes and it helps inform our future development. Textbooks very often provide a timeline of de-bunked ideas/theories before explaining the currently accepted ideas. The changing editions of The Diagnostic and Statistical Manual of Mental Disorders (DSM), and the Atomic Theory come to mind as visible changes made to accepted measures and ideas.

    Diagnostic and Statistical Manual of Mental Disorders. (2013, November 28). In Wikipedia, The Free Encyclopedia. Retrieved 20:38, November 29, 2013, from http://en.wikipedia.org/w/index.php?title=Diagnostic_and_Statistical_Manual_of_Mental_Disorders&oldid=583644464

    Atomic theory. (2013, October 10). In Wikipedia, The Free Encyclopedia. Retrieved 20:37, November 29, 2013, from http://en.wikipedia.org/w/index.php?title=Atomic_theory&oldid=576530542

  2. annastandish

    You quite rightly indicate that it’s silly to act as if by perfecting peer review in isolation, we could fix the problems endemic to scholarly publishing. Sometimes it seems to me that the buggiest part of the system is post-publication, when an article needs to be retracted – and journals don’t always do their best to cooperate (http://retractionwatch.com/2013/11/27/at-long-last-disputed-dance-study-retracted-from-nature/ makes for an interesting read.) As I understand it so far, peer review is only one important piece in a whole assembly line of checks and balances – *but* if that piece can be made more effective, bigger problems can be prevented down the line. (So, to answer your question… a bit of both?)

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s