Thursday, December 13, 2012

Open access, peer review and a Journal of Everything

Open access journals are obviously a good thing for lots of very obvious reasons, and I’m sure they will take over the academic publishing scene at some stage, if only because it’s now so easy to post freely available copies of papers on the web. But there are many other things wrong with the peer reviewed journal system, and the shake up that the spread of open access is likely to prompt may provide opportunities to deal with these too. Those that occur to me at the moment include, in no particular order, the following:

The plethora of different journals makes life difficult for both readers and authors, and must discourage interdisciplinary research which may not conform to the prejudices of any particular specialist area. What is needed is a Journal of everything. There are various public repositories of papers (e.g.  SSRN, which are not peer reviewed, and Sage Open is a peer review journal which spans “the full spectrum of the social and behavioral sciences and the humanities”, but there is as yet no truly general journal. Journals on things like Football Studies made a lot of sense in the days when the devotees of each area received their monthly issue through the post, and could rely on the journal to keep them up to date with everything happening in their area. The web has now changed this: we don’t need research bundling into journals, but instead we simply search for the relevant papers wherever they happen to be. Specialist journals, each with own little idiosyncrasies, are no longer necessary.

Peer review is just what it says it is: review by researchers in the same area, often readers of the same journal. What academic journals don’t get is end user or customer reviews. The predictable result is that published research gets more and more inward looking as authors are forced to conform to the, often arbitrary and unreasonable, prejudices of the particular discipline. This is how Kuhn’s normal science works and it does make sense in established disciplines where the paradigm is a productive one worth pursuing. In areas where there is no such paradigm (management, education?), the conformity enforced by peer reviewers must be bad.

Academic journals usually use two or three peer reviewers for each article, but readers never get to know what the reviewers check for. Are they checking the statistics? The grammar and style? Acknowledgement of previous results? The research design? The usefulness of the research? The originality? Whether the results can be trusted? If an article is on the safety and effectiveness of a new medical treatment, we would probably want to be reassured that the statistics are right, and the conclusions are as trustworthy as possible. If, on the other hand, the idea is to encourage new ideas, not necessarily fully tested, then much looser evaluation criteria would be relevant. But we do need to know whether something is in a journal because the editor and reviewers think it’s right and useful, or because they think it’s an interesting idea which may or may not be worth pursuing. The journal Medical Hypotheses, for example “will publish interesting and important theoretical papers that foster the diversity and debate upon which the scientific process thrives”. Problems only arise (see Wikipedia article) when this sort of review is confused with a review which says an article should be believed.

I think the answer to these problems, and many others, is to establish a Journal of Everything, and then encourage a market in reviewing systems. The Society for Medical Trials might establish a reviewing system and give their stamp of approval to papers which meet their criteria for valid trials of medical procedures. Another body might provide a certificate that a paper is interesting and well written, and another body might check the statistics. In each case their reputation will depend on readers’ accepting their verdicts, and as new requirements arise new reviewing systems would be created to fill the need. I’ve fleshed this out in slightly more detail in an article about the Journal of everything.

No comments: