01 January 2010

For the victims of peer review

Dr Grumble has a research paper on his desk gathering dust. He is fairly sure that its publication would save lives. Unfortunately the reviewers do not agree. The results of Dr Grumble's experiments were so dramatic that Dr G is suspicious that the referees just did not believe them.

The peer review process can be very frustrating. Dr Grumble has frequently had papers reviewed by anonymous 'experts' who revealed their incompetence in their comments. This video says it all.

With thanks to Dr Aust.


Fuddled Medic said...

The laboratory I have been working in has had that video circulating for some time, it raised a few smiles - which for pathologists (or the ones I've seen)means they found it hilarious.

If you want to get it published maybe you should try and make it more trendy? Apparently a common trick is to stick the word "Stem cells" in the title, even if the paper only fleetingly refers to them.

Another trick apparently is to put in a big glaring error so that the referee's see this and comment on it, so that they then go and miss other things

Unlucky though, shit happens

Anonymous said...

This is the best comment you've ever made.

Why don't more doctors think critically about the value of 'peer-reviewed' papers and the politics of publication?

Do med students get any training in how to think critically about such issues?

Happy New Year!

English Pensioner said...

Perhaps you should go in for climate research, they seem to have no problem getting papers peer reviewed, as long as they agree with the general idea.
The idea of having a deliberate mistake is quite good. I used to write technical specifications which were reviewed by the chief engineer before being sent out for tender. He had a "thing" about split infinitives, so I always made a point of putting in a couple for him to find. Worked like magic!

Jobbing Doctor said...

I do peer reviews for a couple of GP-based journals (British Journal of General Practice and Education for Primary Care).

We do not allow anonymous reviews.

Dr Grumble said...

I recently reviewed a paper for an American journal. The paper was from people I knew. Quite probably they recommended me. It had a number of serious flaws. I did my best to be fair. The other reviewers were as harsh as I was.

Within a couple of weeks the paper was back 'on my desk' from a European journal. I wrote to the French editor to explain but got no reply as to what I should do. So I declined to review the paper again because I thought it was fairer for the authors to have a different referee. As it happened I was a uniquely suitable reviewer for the combination of topics in this particular paper so perhaps the right thing to do would have been to put the boot in again. It was my own experience with referees that led me to take the alternative route.

I wonder if this problem could be addressed by borderline (or even hopelessly bad) papers being published online with the referees' critiques alongside.

The amusing video brought home to me something that I knew already but. to use a Rumsfeld-like phrase, didn't know I knew, which is that there are some major problems with peer review which do cause heartache and injustice.

Dr Aust said...

One approach that is gradually gaining popularity is to have on-line comments threads after published papers so that people can post critiques on aspects that, er, "got by" the referees.

Interested to hear from JD that the GP journals have de-anonymised their peer review. There are a very few journals in the biosciences that have "open" review (the well-regarded specialist journal Cell Calcium is one I know about) but on the whole they remain anonymous. Though of course if a reviewer asks you to put in citations to four of their papers then they are not all that anonymous.

Did you consider just sending the European journal the review you had done for the American one, Dr G? I think I might have done that and said: "Here it is, done for x journal. Of course, you may prefer a different reviewer..." Although reading it I guess that is near enough what you did.

Dr Grumble said...

The problem, Dr Aust, was that I couldn't look at the new version without clicking to accept the invitation to review. It was possible that they had addressed all my previous points but extremely unlikely owing to the rapidity of it reaching the other journal and flaws too fundamental to rectify. I did my best to resolve my doubts about what to do by emailing the editor but either he did not bother to reply or it was one of those automated things that would not receive a reply. Anyway I was very busy at the time and I thought it was fair to give the authors another reviewer.

Another thing that irritates me about all of this is that the authors are paid nothing and the referees are paid nothing but the copyright goes to the publishers and viewing the paper may require payment of a charge which goes to publishers that we barely need.

I have thought of just plonking my unpublished paper on the web for anybody who is interested to find. This would cost me nothing at all, the paper would be free to all who wished to read it, I could allow comments (which I agree is an excellent approach) and I could keep the copyright.

Fuddled Medic said...

By publishing it on the internet for free of charge, you may save other researchers time and effort. it may help them with setting up there experiments, finding suitable subjects, ordering the right material etc.

Dr Aust said...

Yes, I have been through the annoyance of finding I can't read one of my own published works online, Dr G. I guess it is a pretty much universal experience among those publishing in scientific and medical journals.

As you will see if you check the comments on my blogpost, my friend Prof David Colquhoun seems to be getting towards the view you express at the end, i.e. of just sticking stuff on a website and opening it up for comments.