Peer review process
Posted: Sat Dec 08, 2007 2:35 pm
(This is somewhat off-topic from the original thread "EBV and MS" so I moved it out.)
Bob,
Regarding the peer review process: Arrr!!! Me thinks you have some misgivings about the pirates' code!
Some might think of it as a necessary evil, but overall it works well. What would be the alternatives? You have a boiling pot of good and bad science, big egos versus bigger egos, lots of time and careers invested with publication records being very important in the funding of future grants and a researcher's survival at stake.
The way the peer review process works typically is that the editors will ask 2-4 researchers in similar areas to read and critique the manuscript. Sometimes the reviewers are suggested by the authors, sometimes the authors will let the editors know of researchers who should not be included because of potential conflicts. Reviewers should let the editors know of any conflicts they might have in becoming involved. Usually the reviewers are kept anonymous from the authors' perspective. Some research areas are so small that it is often fairly obvious to the authors who reviewed their manuscript.
The reviewers typically have 2 weeks or so to read the manuscript, check reference articles listed and then submit their review. Their review should give a synopsis of the manuscript to show, in their own words, that they grasp the concepts. They should mention any points that need correcting or refinement. And the review will rate the manuscript on its importance and scientific merit, recommend whether or not the reviewer thinks it should be published, and whether there are major or minor revisions needed before the reviewer deems it worthy of publishing. It is rare for a paper to be accepted without reviewers wanting at least minor revisions. It is then up to the authors to make the requested changes, make a successful refutation as to why the changes are not needed, or withdraw their manuscript and try submitting it to a less important journal.
The process can be abused. A manuscript may be rejected and within the year a competitor, who may have been a reviewer of the first manuscript, gets something quite similar published. Did he/she steal the ideas, intentionally blackballing the original manuscript? That's possible. I've felt that happened to me once but life goes on. I doubt it occurs often.
There is the possibility that a reviewer does not agree with some underlying premise and harps on some reason to recommend not publishing the manuscript. You will probably see the authors writing a letter of refutation that is as long as or longer than their original manuscript because their ego kicks in when they see the reviewers' comments. That is why it is good to have multiple reviewers to avoid the spats. The editors of course are the final decision makers. The editors can also develop a positive or negative opinion of given authors over the years. In some tight-knit areas, editors (who are also authors and researchers) could potentially get too cozy with some authors.
For the most part, I think reviewers are fair and stick to logic to express their reservations about a manuscript. With the number of manuscripts that editors receive at prestigious journals, they are looking for reasons not to publish something so it is best for the authors to give their very best efforts in preparing the first submission.
I've been a reviewer (even on one or two of the articles mentioned in this forum). I have my own ideas about autoimmune diseases but I have to put that aside and review the manuscripts for the soundness of the work done and the conclusions that the authors make from their own work. I always ask for at least minor revisions and it is satisfying to see that they are incorporated. Reviewers usually get to see each others comments and the reviewers usually harp on the same points of concern.
Getting the reviews back can be a really maddening event for the authors or it can be a really big boost to the ego, depending on the ratings and comments.
One thing that is controversial is the authorship. Depending on the area (chemistry, biology, physics, etc.) the journal, and the priniciple investigator's ego, the order of authors and the listing of people as authors is very important. Usually, at least in biology, the person who did most of the work is listed first. He/she spent the long hours getting the experiments to work, perhaps input most of the technical ideas, and did most of the writing of drafts for the manuscript. They are perhaps the one working on a PhD or getting ready to apply for a grant. They are the one who needs the publication the most. The last author is usually the principle investigator in whose lab the work was done and who funded the work. Sometimes the big shot wants to be listed first since all future citations will be in the form 'Smith, et al.'. The other authors listed on the manuscript should have had some significant effort on the work. Fortunately many journals are now asking for submission of a form describing the contributions of each author to the manuscript. (Having an affair with one of the other authors or editors should not qualify!!!...say no more).
There is another process you alluded to: follow up discussion of published articles. This is usually the area of the 'Letters to the Editor'. After an article appears, you can write a short letter discussing a point with which you disagree or on which you want to elaborate. These letters also go through some review and the authors of the original article are usually asked to provide a response to the letter so the letter and response can appear in the same issue. Because of this, 'Letters to the Editor' may come three months to a year after the original article so they do not seem quite so timely. If it is a bigshot researcher and the area is hot, it could make it into the next issue.
Why do researchers agree to be reviewers? They can get some insight into what is going on in their area or related areas before the rest of the research community. They can also protect their own projects by rushing to publication something from their lab that is similar to a manuscript they are reviewing. In that case, they should notify the editor who may decide to publish papers from the two labs in the same issue. Many mentors will do reviewing to protect the projects of their graduate students, who really need a couple of original papers published to qualify for graduation. Getting scooped on something they spent 5-6 years on is really disheartening. Also reviewers are always trying to keep their sharpness at assessing good and bad science in order to improve their own writing skills, for manuscripts and grant applications. Maybe they think an editor will have some leniency towards publishing their articles if they help review others. That would certainly be getting into a gray area of ethics though. Personally I do it because I think it helps me in being thorough when I write my own manuscripts.
Peer review of grants is another animal, one with which I have little experience. There the reviewers get a lot of grants (maybe 20) they have to study in minute detail and compare. Then they have to meet and, as a committee, discuss and rate and prioritize the grants for the limited funding. Their recommendations then go to a funding review group that makes the final decision based on ratings of grant applications, available funds, and preferred areas of research according to guidelines from NIH higher-ups or whoever. Typically only about 10-15% of grants get funded. Manuscripts get accepted for publication at perhaps a 20-25% rate, depending on the journal. Some journals are tougher and will not even accept manuscripts without sponsership by a member of a prestigious science association.
Wesley
Bob,
Regarding the peer review process: Arrr!!! Me thinks you have some misgivings about the pirates' code!
Some might think of it as a necessary evil, but overall it works well. What would be the alternatives? You have a boiling pot of good and bad science, big egos versus bigger egos, lots of time and careers invested with publication records being very important in the funding of future grants and a researcher's survival at stake.
The way the peer review process works typically is that the editors will ask 2-4 researchers in similar areas to read and critique the manuscript. Sometimes the reviewers are suggested by the authors, sometimes the authors will let the editors know of researchers who should not be included because of potential conflicts. Reviewers should let the editors know of any conflicts they might have in becoming involved. Usually the reviewers are kept anonymous from the authors' perspective. Some research areas are so small that it is often fairly obvious to the authors who reviewed their manuscript.
The reviewers typically have 2 weeks or so to read the manuscript, check reference articles listed and then submit their review. Their review should give a synopsis of the manuscript to show, in their own words, that they grasp the concepts. They should mention any points that need correcting or refinement. And the review will rate the manuscript on its importance and scientific merit, recommend whether or not the reviewer thinks it should be published, and whether there are major or minor revisions needed before the reviewer deems it worthy of publishing. It is rare for a paper to be accepted without reviewers wanting at least minor revisions. It is then up to the authors to make the requested changes, make a successful refutation as to why the changes are not needed, or withdraw their manuscript and try submitting it to a less important journal.
The process can be abused. A manuscript may be rejected and within the year a competitor, who may have been a reviewer of the first manuscript, gets something quite similar published. Did he/she steal the ideas, intentionally blackballing the original manuscript? That's possible. I've felt that happened to me once but life goes on. I doubt it occurs often.
There is the possibility that a reviewer does not agree with some underlying premise and harps on some reason to recommend not publishing the manuscript. You will probably see the authors writing a letter of refutation that is as long as or longer than their original manuscript because their ego kicks in when they see the reviewers' comments. That is why it is good to have multiple reviewers to avoid the spats. The editors of course are the final decision makers. The editors can also develop a positive or negative opinion of given authors over the years. In some tight-knit areas, editors (who are also authors and researchers) could potentially get too cozy with some authors.
For the most part, I think reviewers are fair and stick to logic to express their reservations about a manuscript. With the number of manuscripts that editors receive at prestigious journals, they are looking for reasons not to publish something so it is best for the authors to give their very best efforts in preparing the first submission.
I've been a reviewer (even on one or two of the articles mentioned in this forum). I have my own ideas about autoimmune diseases but I have to put that aside and review the manuscripts for the soundness of the work done and the conclusions that the authors make from their own work. I always ask for at least minor revisions and it is satisfying to see that they are incorporated. Reviewers usually get to see each others comments and the reviewers usually harp on the same points of concern.
Getting the reviews back can be a really maddening event for the authors or it can be a really big boost to the ego, depending on the ratings and comments.
One thing that is controversial is the authorship. Depending on the area (chemistry, biology, physics, etc.) the journal, and the priniciple investigator's ego, the order of authors and the listing of people as authors is very important. Usually, at least in biology, the person who did most of the work is listed first. He/she spent the long hours getting the experiments to work, perhaps input most of the technical ideas, and did most of the writing of drafts for the manuscript. They are perhaps the one working on a PhD or getting ready to apply for a grant. They are the one who needs the publication the most. The last author is usually the principle investigator in whose lab the work was done and who funded the work. Sometimes the big shot wants to be listed first since all future citations will be in the form 'Smith, et al.'. The other authors listed on the manuscript should have had some significant effort on the work. Fortunately many journals are now asking for submission of a form describing the contributions of each author to the manuscript. (Having an affair with one of the other authors or editors should not qualify!!!...say no more).
There is another process you alluded to: follow up discussion of published articles. This is usually the area of the 'Letters to the Editor'. After an article appears, you can write a short letter discussing a point with which you disagree or on which you want to elaborate. These letters also go through some review and the authors of the original article are usually asked to provide a response to the letter so the letter and response can appear in the same issue. Because of this, 'Letters to the Editor' may come three months to a year after the original article so they do not seem quite so timely. If it is a bigshot researcher and the area is hot, it could make it into the next issue.
Why do researchers agree to be reviewers? They can get some insight into what is going on in their area or related areas before the rest of the research community. They can also protect their own projects by rushing to publication something from their lab that is similar to a manuscript they are reviewing. In that case, they should notify the editor who may decide to publish papers from the two labs in the same issue. Many mentors will do reviewing to protect the projects of their graduate students, who really need a couple of original papers published to qualify for graduation. Getting scooped on something they spent 5-6 years on is really disheartening. Also reviewers are always trying to keep their sharpness at assessing good and bad science in order to improve their own writing skills, for manuscripts and grant applications. Maybe they think an editor will have some leniency towards publishing their articles if they help review others. That would certainly be getting into a gray area of ethics though. Personally I do it because I think it helps me in being thorough when I write my own manuscripts.
Peer review of grants is another animal, one with which I have little experience. There the reviewers get a lot of grants (maybe 20) they have to study in minute detail and compare. Then they have to meet and, as a committee, discuss and rate and prioritize the grants for the limited funding. Their recommendations then go to a funding review group that makes the final decision based on ratings of grant applications, available funds, and preferred areas of research according to guidelines from NIH higher-ups or whoever. Typically only about 10-15% of grants get funded. Manuscripts get accepted for publication at perhaps a 20-25% rate, depending on the journal. Some journals are tougher and will not even accept manuscripts without sponsership by a member of a prestigious science association.
Wesley
