Recently I happened to see a reviewer's comments on a manuscript submitted to a good journal (not my manuscript, and I didn't help review it either). The review, heavily paraphrased, went something like this:
"The authors show evidence in favor of a strange result.
"The evidence per se isn't bad; however, many respected labs do the same type of technique as these authors, and the respected labs don't see these results.
"Ergo, Strange Result is likely the result of inexperience or idiocy on the authors' part.
"Vote to reject."
I am of mixed feelings about this. On the one hand, this is the sort of dismissive attitude that makes it difficult for revolutionary research to get published in good journals. It contributes to the do-safe-research-with-expected-results mindset: if your research turns up something too novel, you can't even publish it!
On the other, I think the reviewer might be right.
15 years ago
10 comments:
I think the problem is that the two cases are often indistinguishable!
Yep. Hard to know unless you're familiar with the technique and ALL the gory details of said precedent as well as current case in point.
Usually you're not privy to all those details in a paper, because the current practice is to not show all the controls and pilot experiments and methodological details.
Methods sections are for suckers, right?
It is a difficult situation. I was at a meeting when Roderick MacKinnon (Nobel Prize winner for crystal structure of K channels) spoke. He structure refuted decades of data and he was not well received. What if he data was not published? where would we be?
I think that if you come up with an "unusual" result, then it may behoove you to back it up using another method. I know that when Advisor reviews a paper like this he'll usually reject/ask for an experiment to back up the strange result. That may be a solution to the problem.
I recently published a paper that said weird things about a certain behavior that people had a general understanding of but not many details. I took it upon myself to cover all the potential sources of variation and provide extra data and discussion in the supplementary information to show that what I was seeing was not any obvious artifact. My reviewers asked for a little more, including more of the data in the actual text, and that was fine--and I was fortunate I guess that they agreed it showed it as well as could be shown.
In that situation, it's on the authors to really prove what they are trying to show. Maybe like you doubt they hadn't justified their claims.
What Amanda and Arlenna said...
I agree that, wherever possible, another experiment using a different method can only help test the hypothesis. But sometimes it's not possible or reasonable to require that the authors do everything at once. In my field, the 'everything at once' attitude is prevalent and from what I can tell, it makes the science very shallow. Lots of people using lots of techniques, but none of them are really used correctly.
and arlenna,
what you're saying falls in the category of "guilty until proven innocent."
Stories like this make me wonder if we shouldn't just let people publish what they have, even if it totally disagrees with what we expect.
WHICH journal should publish it is a totally different problem.
But MsPhD, it's just good science to fully demonstrate whatever it is you are trying to claim, no matter what the context. Whether that is something people have seen before or not, it's one of the fundamental tenets of recording and communicating scientific work.
It's not about guilt or innocence, it's about justifying one's interpretations and opinions in ways that fully communicate it to other people. Normally that means fleshing out every control for artifact that you possibly can. If people can't do that, why should we let them say "Hey, even though I can't prove part X to you, just believe me because I am so nice or so famous or trying so hard or whatever else."
Arlenna, I think the issue is that your version puts giant burden on the scientist's resources. A scientist at a Big Fancy Institution is likely to have access to more useful tools for supporting their hypothesis. So a creative scientist at a small place can end up getting shut out of the publication debate if their data conflict. But it's true that the more extraordinary your claim, the more extensive the evidence should be.
I would note that in this instance, the group's strange finding had already been published by a Fancy Group in a Glamor Mag with no extra verification. So this manuscript was a "we see it too!" and yet got rejected at a lower journal for lack of evidence. Given that this group was not a FancyLab, it smacks of different standards for famous people.
That sounds lame, to push back something saying "We see it too," when it was the 2nd report of it even if it was unusual overall.
In my field (which covers quite a lot of biology and chemistry), there's very little for which you couldn't show all the right controls with normal everyday experiments, so it just seems weird to me that controls covering the potential sources for artifact couldn't be done by whoever needed to do them. Would that be the case for work done with things like mouse models (where you have to develop the right mice over the course of like 5 years) or synchrotrons (where every experiment costs like 5 million dollars) or things like that?
Post a Comment