Open access journals are obviously a good thing for lots of
very obvious reasons, and I’m sure they will take over the academic publishing scene
at some stage, if only because it’s now so easy to post freely available copies
of papers on the web. But there are many other things wrong with the peer
reviewed journal system, and the shake up that the spread of open access is
likely to prompt may provide opportunities to deal with these too. Those that
occur to me at the moment include, in no particular order, the following:
The plethora of different journals makes life difficult for both
readers and authors, and must discourage interdisciplinary research which may
not conform to the prejudices of any particular specialist area. What is needed
is a Journal
of everything. There are various public repositories of papers (e.g. SSRN, arXiv.org) which are not peer reviewed, and Sage Open is a peer review journal which spans
“the full spectrum of the social and behavioral sciences and the humanities”,
but there is as yet no truly general journal. Journals on things like Football Studies
made a lot of sense in the days when the devotees of each area received their
monthly issue through the post, and could rely on the journal to keep them up
to date with everything happening in their area. The web has now changed this:
we don’t need research bundling into journals, but instead we simply search for
the relevant papers wherever they happen to be. Specialist journals, each with
own little idiosyncrasies, are no longer necessary.
Peer review is just what it says it is: review by
researchers in the same area, often readers of the same journal. What academic
journals don’t get is end user or customer reviews. The predictable result is
that published research gets more and more inward looking as authors are forced
to conform to the, often arbitrary and unreasonable, prejudices of the particular
discipline. This is how Kuhn’s normal science works and it does make sense in
established disciplines where the paradigm is a productive one worth pursuing.
In areas where there is no such paradigm (management, education?), the
conformity enforced by peer reviewers must be bad.
Academic journals usually use two or three peer reviewers
for each article, but readers never get to know what the reviewers check for.
Are they checking the statistics? The grammar and style? Acknowledgement of
previous results? The research design? The usefulness of the research? The
originality? Whether the results can be trusted? If an article is on the safety
and effectiveness of a new medical treatment, we would probably want to be
reassured that the statistics are right, and the conclusions are as trustworthy
as possible. If, on the other hand, the idea is to encourage new ideas, not
necessarily fully tested, then much looser evaluation criteria would be
relevant. But we do need to know whether something is in a journal because the editor
and reviewers think it’s right and useful, or because they think it’s an interesting
idea which may or may not be worth pursuing. The journal Medical Hypotheses,
for example “will publish interesting and important theoretical papers that
foster the diversity and debate upon which the scientific process thrives”.
Problems only arise (see Wikipedia article)
when this sort of review is confused with a review which says an article should
be believed.
I think the answer to these problems, and many others, is to
establish a Journal of Everything, and then encourage a market in reviewing
systems. The Society for Medical Trials might establish a reviewing system and
give their stamp of approval to papers which meet their criteria for valid
trials of medical procedures. Another body might provide a certificate that a
paper is interesting and well written, and another body might check the
statistics. In each case their reputation will depend on readers’ accepting their
verdicts, and as new requirements arise new reviewing systems would be created
to fill the need. I’ve fleshed this out in slightly more detail in an article
about the Journal
of everything.
No comments:
Post a Comment