There is an interesting debate about "peer review" going on among readers, contributors and editors of "Nature", the scientific journal. There is also some activity on the part of interested bloggers (see, for example, here and here). The debate and the lengths to which "Nature" is changing its processes is another leading indicator of how much IT is changing how people work.
Background
Nature is a "refereed scientific journal," meaning articles are reviewed by others in the same field and they must approve the article before the editors will (consider) publishing the piece. This time-honored tradition, involving mailing physical copies of the research to reviewers and exchanging comments and recommendations (often anonymously) through the editor, replaced the earlier tradition of live debate in scientific societies. IT and the social collaborations that flourish on the Internet enable scientists to try "open debate" instead of closed (and confidential) peer review.
"Nature" is an interesting player in this arena. In November and December 2005, there was a large (some say manufactured) controversy surrounding the quality of content on Wikipedia. The controversy flared over the "Sigenthaler incident." "Nature" ran a special report that commented on the quality of science coverage on Wikipedia compared with the quality of articles found in the Encyclopedia Britannica. It concluded that "Jimmy Wales' Wikipedia comes close to Britannica in terms of the accuracy of its science entries." Britannica disagreed and "Nature" carried that - and its response to Britannica's position - as links found at the beginning of the special report. ("The Wall Street Journal" recently ran a debate between the founder of Wikipedia and the editor of Britannica. See this link for the full article.) And Gartner wrote a research note on lessons to be learned by corporate IT managers from the Wikipedia controversy.
Today
Fast forward to this summer. "Nature" has opened a debate on how to change peer review. Here's its description of the debate (see this link for the full article):
"Nature's" Peer Review Debate
"Peer review is commonly accepted as an essential part of scientific publication. But the ways peer review is put into practice vary across journals and disciplines. What is the best method of peer review? Is it truly a value-adding process? What are the ethical concerns? And how can new technology be used to improve traditional models?
This "Nature" Web debate consists of 22 articles of analyses and perspectives from leading scientists, publishers and other stakeholders to address these questions."
Nature also set up a separate space for comments on the debate and is making unapproved research available for comment (by researchers in the subject field). This peer review blog is its new experiment.
Finally, there's an interesting paper on the benefits of using the "wisdom of crowds" to improve the quality of peer review. "Nature's" debate on the nature of peer review can be found at here.
"Nature" isn't the only journal experimenting with peer review, but its involvement in the Sigenthaler incident, as well as its high overall profile, makes these experiments interesting.
How does all of this apply to your internal and external technical strategies? How would you like us to follow up on this blog posting with more-formal research?
While "Nature" and others may find 'peer-review' in need of fixing, I question the fundamental premise of the need for change.
The concept of peer-review ensures that those who are knowledgeable about a subject have responsibly examined the work of colleagues in an attempt to produce illuminated and interesting new information for those interested in a subject. Peer review is unnecessary in journalism per se, although the practice is actually not uncommon in the editorial process.
What perhaps is most of concern is the allusion that a Wiki, open to most all potential contributors without portfolio, becomes a point of reference and even trusted by those less informed about a key topic. While this might appear insignificant on the surface, consider that knowledge is a compendium of established facts (and suspicions) constantly changing in importance to various factions.
What if suddenly there were no trustworthy source of information? How would this impact the ability of science to proceed? Experiments that had been validated through peer review would have to be redone simply to assure the outcomes were reasonable and that protocols were sound.
There is no replacement for peer review in science. Perhaps in other less critical areas where peer review has been an accepted practice in the past it could be now retired. But in the practices of formal sciences, peer review serves as an important and necessary control against complete chaos.
