Results-free review: a new model of peer review
Table of contents
Openness and transparency are the pillars of scientific progress. However, there is a bias towards publishing positive results, distorting the evidence base and undermining the reproducibility of research. Results-free review offers a solution to this problem, focusing editorial decisions on the rationale and methods alone.
What is results-free review?
Results-free review is a new model of peer review, where editors and reviewers are blinded to the results. It happens in two stages:
Stage 1: Review of manuscript, excluding any results or discussion of results.
Stage 2: If accepted following Stage 1, review of complete manuscript to check the results and conclusions do not deviate unjustifiably from the research question and methodology.
How is it different to normal peer review?
All aspects of the peer-review process are the same (with the exception of Stage 1), including the questions for reviewers, criteria for publication and expected turnaround times. See our peer-review policies for more details.
How is it different to Registered Reports?
In Registered Reports, peer review of the study protocol is completed before any experiments are conducted. In results-free review, peer review occurs after the experiments are conducted, as is the case for traditional post-study peer review. We support the Registered Reports format and consider both approaches as complementary.
Your contribution to the future of peer review
The number of new publishing initiatives is on the rise. It is important that these are rigorously evaluated to ensure they are having the desired outcomes. In choosing results-free review, you are also contributing to our randomised-controlled trial in determining how it influences publication bias and the editorial decision-making process. If deemed effective, it is our hope that results-free review will become the ‘norm’ for peer review.
The editorial workflow of BMC Psychology's pilot of results-free peer review is summarized in this flowchart.
For research articles which are part of the results-free peer review pilot (results-free research articles) the authors have been asked to remove all results and discussion from their research article prior to editorial assessment and peer-review. If the methodology is deemed to be suitable and appropriate following initial peer-review, the authors will be asked to include their results and discussion in a full manuscript. These articles will be indicated by a red R symbol underneath the Action list for the manuscript.
Editorial assessment of a results-free pilot manuscript
During the initial ‘results-free’ stage of peer-review key points to assess are:
✔ Is there a scientifically valid research question?
✔ Are the methods and analysis suitable and appropriately applied?
✔ Is sufficient detail provided to assess the soundness of the methodology?
✔ Have any appropriate community standards which apply to this research field or study type been followed?
Once the methodology of the ‘results-free’ manuscript has been deemed to be rigorous, the second stage of peer-review will assess the full manuscript. During this second stage, key points to assess are:
✔ Are the conclusions appropriate to the methods used?
✔ Are the conclusions supported by the results presented?
Please remember that editorial decisions should NOT be made on the basis of the interest of a study or its likely impact.
Editorial decisions for a results-free pilot manuscript
Overview of NEW types of editorial decisions for results-free peer review
The following are NEW decisions types which are ONLY for use when making a first decision regarding a manuscript which is taking part in the trial of results-free peer-review.
• Accept in principle: This decision should be used if the rationale and methodology are deemed to be suitable and appropriate – the authors will be asked to include their results and discussion in the revised manuscript and to address small essential revisions or clarifications.
• Decision not possible: This decision should be used for cases where a lack of detail does not allow sufficient assessment of the rationale and methodology for a decision regarding validity to be reached – the authors will be asked to submit a revised result-free manuscript.
Please DO NOT submit an editorial decision of either ‘minor revisions’ or ‘major revisions’ when making the first decision on a ‘results-free’ manuscript. Theses decision terms can still be used for manuscripts which are not taking part in the ‘results-free’ peer review trial.
Overview of other types of editorial decisions for results-free peer review:
• Reject before/after review (and invite resubmission): For major revisions that will take longer than 3 months to complete. The file is closed and the authors invited to resubmit.
• Reject after review: To be used only if the study’s methodology is fundamentally flawed and cannot be revised. Resubmission is not invited.
• Accept – If no further revisions are required, and you are satisfied that the work is scientifically valid.
Providing comments to authors
We strongly encourage editors to include their own comments to authors when forming editorial decisions. The handling editor should provide feedback that’s constructive and fair. Where appropriate, please provide advice regarding the essential revisions to allow the authors to understand and address exactly is required to get their work published.
Checklist for editorial decisions
✔ Is the decision you are making in accordance with the threshold of accepting all scientifically valid research?
✔ Are you satisfied with the scientific assessment of the article?
✔ Are the reviewers’ comments to the authors constructive and fair?
✔ Have you, or the reviewers been able to assess the statistical reporting?
✔ Are there two reviewer reports?
✔ If there are any methodological concerns, could the authors revise the manuscript to address these?
Further information is available for our editors here - http://www.biomedcentral.com/foreditors
BMC Psychology is participating in a pilot study of Results-free peer review. ‘Results-free’ means that editors and reviewers will be blinded to the study’s results during the initial stage of peer review. Research articles will be primarily assessed based on the scientific merits of the rationale and soundness of the methods alone, in the absence of results or discussion sections, which will be included after the initial peer-review of the results-free manuscript.
This guide for reviewers contains information about basic considerations that should be applied when reviewing a ‘results-free’ manuscript that has been submitted to BMC Psychology, and about the editorial standards of the journal. Other relevant information about the journal’s aims and scope and editorial policies can be found on the 'About' page.
Submitted manuscripts are usually reviewed by two or more experts. Peer reviewers will be asked to recommend whether a manuscript should be accepted, revised or rejected. They should also alert the editors of any issues relating to author misconduct such as plagiarism and unethical behavior.
BMC Psychology operates using an open peer review system, where the reviewers' names are included on the peer review reports. In addition, if the article is published, the named reviewer reports are published online alongside the article under a Creative Commons Attribution License 4.0. Pre-publication versions of the article and author comments to reviewers are available upon request.
Publication of research articles by BMC Psychology is dependent primarily on their validity and coherence, as judged by peer reviewers and editors. BMC Psychology do not make editorial decisions on the basis of the interest of a study or its likely impact. Studies must be scientifically valid; for research articles this includes a scientifically sound research question, the use of suitable methods and analysis, and following community-agreed standards relevant to the research field. Submitted manuscripts will be sent to peer reviewers, unless they are out of scope or if the presentation or written English is of an unacceptably low standard.
Points to consider
For research articles which are part of the results-free peer review pilot (results-free research articles) the authors have been asked to remove all results and discussion from their research article prior to editorial assessment and peer-review; this includes the abstract. If the methodology is deemed to be suitable and appropriate following initial peer-review, the authors will be asked to include their results and discussion in a full manuscript.
Reviewers are asked to provide detailed, constructive comments that will help the editors make a decision on publication and the author(s) improve their manuscript. A key issue is whether the work has serious flaws that should preclude its publication. Where possible, reviewers should provide references to substantiate their comments.
Reviewers of ‘results-free’ manuscripts should address the points below and indicate whether they consider any required revisions to be 'major compulsory revisions', 'minor essential revisions' or 'discretionary revisions'. In general, revisions are likely to be 'Major compulsory revisions' if additional controls are required, or if the methods used are inadequate or statistical errors have been made.
Key points to assess for a ‘results-free’ manuscript are as follows:
✔ Is the question posed original, important and well defined?
The research question posed by the authors should be easily identifiable and understood.
It is useful to both the editors and authors if reviewers comment on the originality and importance of the study within the context of its field. If the research question is unoriginal because related work has been published previously, please give references.
✔ Are the methods appropriate and well described, and are sufficient details provided to allow others to evaluate and/or replicate the work?
Please remark on the suitability of the methods for the study, which should be clearly described and reproducible by peers in the field.
If statistical analyses have been carried out, specify whether or not they need to be assessed specifically by an additional reviewer with statistical expertise.
✔ Are the appropriate controls included?
If you feel that inappropriate controls have been used please say so, indicating the reasons for your concerns, and suggesting alternative controls where appropriate.
✔ What are the strengths and weaknesses of the methods?
Please comment on any improvements that could be made to the study design to enhance the quality of the study. If novel experimental techniques were used please pay special attention to their reliability and validity.
✔ Are there any ethical or competing interests issues you would like to raise?
The study should adhere to ethical standards of scientific/medical research and the authors should declare that they have received ethics approval and or patient consent for the study, where appropriate.
Whilst we do not expect reviewers to delve into authors' competing interests, if you are aware of any issues that you do not think have been adequately addressed, please inform the editorial office.
✔ Reviewers are reminded of the importance of timely reviews.
If reviewers encounter or foresee any problems meeting the deadline for a report, they should contact firstname.lastname@example.org.
Any manuscript sent for peer review is a confidential document and should remain so until it is formally published.
Once the methodology of the ‘results-free’ manuscript has been deemed to be rigorous during the first round of peer-review, the second stage of peer-review will assess the full manuscript.
During this second stage, key points to assess are:
✔ Is the interpretation (discussion and conclusion) well balanced and supported by the data?
The interpretation should discuss the relevance of all the results in an unbiased manner. Are the interpretations overly positive or negative?
Conclusions drawn from the study should be valid and result directly from the data shown, with reference to other relevant work as applicable. Are the conclusions appropriate to the methods used and the results presented? Have the authors provided references wherever necessary?
✔ Can the writing, organization, tables and figures be improved?
Although the editorial team may also assess the quality of the written English, please do comment if you consider the standard is below that expected for a scientific publication.
If the manuscript is organized in such a manner that it is illogical or not easily accessible to the reader please suggest improvements.
Please provide feedback on whether the data are presented in the most appropriate manner; for example, is a table being used where a graph would give increased clarity? Do the figures appear to be genuine, i.e. without evidence of manipulation, and of a high enough quality to be published in their present form?
✔ When revisions are requested.
Reviewers may recommend revisions for any or all of the following reasons: data need to be added to support the authors' conclusions; better justification is needed for the arguments based on existing data; or the clarity and/or coherence of the paper needs to be improved.
✔ Are the included additional files (supplementary materials) appropriate?
Online publishing enables the inclusion of additional files with published articles. Additional files of many types can be submitted, including movies, tabular data and mini-websites. Reviewers are encouraged to comment on the appropriateness of the types of additional files, included with the manuscript, for publication with the final article. Additional files pertaining to original/raw data files that support the results reported in the manuscript can be included. It is not expected that reviewers should reanalyze all supporting data as part of their peer review, but the availability of supporting data enables more detailed investigation of particular aspects of the study if the reviewer or editor feels it is necessary.
Portability of peer review
To support efficient and thorough peer review, we aim to reduce the number of times a manuscript is reviewed, thereby speeding up the publication process and reducing the burden on peer reviewers. Therefore, please note that, if a manuscript is not accepted for publication in BMC Psychology and the authors choose to submit a revised version to another BioMed Central published journal, we will pass the reviews on to the other journal's editors at the authors’ request. We will reveal the reviewers' names to the handling editor for editorial purposes unless reviewers let us know when they return their report that they do not wish us to share their report with another BioMed Central published journal and/or that they do not wish to participate further in the peer review of this manuscript.
Reviewers are asked to bear the editorial standards of BMC Psychology in mind and alert the editors if authors have not fully adhered to them. BMC Psychology is a member of the Committee on Publication Ethics (COPE).
Standards of reporting
BMC Psychology supports initiatives aimed at improving the reporting of research. Reviewers can find more details on this at Standards of Reporting.
How do I log in?
When you are invited to review a manuscript, you will receive an email with the manuscript details and abstract, and links to either agree or decline to review. By clicking on either of these links, you will be taken to the journal website and automatically logged in. You will be able to access the manuscript file, if you agreed to review, and then upload your report.
How do I use the tools?
After you have agreed to review, you will see the link to your pending assignments. Clicking this will take you to a list of all the manuscripts in which you are involved as a reviewer on that journal. From here you can download the manuscript file (“View Manuscript”) and then create and submit your report (“Submit recommendation”). Clicking the link to submit your report will take you to the online form to create your report. There will be some standard questions about language quality etc., and you will be asked to complete a declaration of competing interests. Thereafter you will be able to include your comments to the authors and confidential comments to the Editors.
Please note that you can save your report and return to it later before submitting, and you can also print a copy of your report for your own records.
What if the manuscript is written too poorly for me to understand?
Usually, this would be identified by the Editor prior to inviting reviewers. However, if you feel unable to review a manuscript for this reason, please contact the Editorial Office as soon as possible. Reviewers are not expected to rewrite manuscripts, and authors may be asked to improve the quality of written English before peer review resumes.
Can I share the manuscript with someone else for help?
Manuscripts are privileged communications and should not be shared with anyone else without first seeking permission from the Editor. However, the Editor may well be happy for you to share with a colleague such as someone with additional expertise that would help the review process. The Editor may also allow you to share the manuscript with a junior colleague as part of peer reviewer training.
What is considered a competing interest?
A competing interest exists when a reviewer’s personal or financial relationship with other people or organisations may unduly influence their review of a manuscript. We ask all reviewers to declare any potential financial or non-financial competing interests. Non-financial competing interests may include personal, intellectual, professional, political or religious competing interests. If you have a competing interest your review may be excluded from the process. If you are unsure whether you have a competing interest, please see our editorial polices page or consult the Editorial Office as soon as possible.
What if I have already reviewed the manuscript for another journal?
You may be sent a manuscript that you have reviewed previously for another journal. For most manuscripts there is no reason why you should not act as a reviewer again. However, if you are asked to review a ‘results-free’ manuscript that you have reviewed previously for another journal we ask that you please let the Editorial Office know, and a new reviewer will be invited. This is because you will have already reviewed the results of the manuscript for the other journal.
In which situations should I decline to review a manuscript?
You may be inadvertently sent a manuscript that it would not be appropriate for you to review. In general you should not review a manuscript if:
• You do not have the necessary expertise to be able to assess the work
• The work is so similar to your own that you would be unable to be objective in your review, or reviewing would compromise your own work significantly.
• Any of the authors are at your institution
• You have an association with any of the authors, e.g. you have a close collaboration or you have been asked to comment on the manuscript before submission
• You would not want the authors to review your work or you have previously requested that they be excluded from reviewing your work
• You have a significant competing interest with regards to the manuscript or subject.
• You have previously reviewed a ‘results-free’ manuscript for another journal.
What if I think the manuscript is outside the scope of the journal?
Please contact the Editorial Office as soon as possible to explain why you do not think the manuscript fits the scope of the journal. If the Editor agrees, they may be able to suggest a better journal option to the author. Alternatively, they may be able to explain why they consider it to be in scope, and reassure you that it is fine to proceed with a normal review.
What is open peer review?
For journals operating an open peer review process, reviewers' names are included on the peer review reports that are sent to the authors. In addition, if the article is published, the named reports are published online alongside the article as part of a 'pre-publication history'. Author responses to the reviewers are also available to readers. Please note that confidential comments to the Editor are not made available to the authors or published alongside the manuscript.
Writing your report
What should I do if I have any ethical concerns?
If you have any concerns about the ethics of animal or human research, the originality of the work, the integrity of the data presented or any potential undeclared conflicts of interest of the authors please outline these in the ‘Confidential Comments to the Editor’ section of your report or contact the Editorial Office.
What if I have concerns about the statistics?
If you are unable to assess the statistics in a manuscript, please note this clearly in your report and indicate whether you think that the manuscript should be seen by an expert statistical advisor. If you have specific concerns regarding the statistics, please detail these in your report. If there is a particular area where you feel expert statistical advice is required, please indicate this in the ‘Confidential Comments to the Editor’ section of your report.
How should I structure my comments to the authors?
It helps to begin with a brief introductory statement summarising the work; if you feel that the work is particularly interesting or novel the authors will appreciate such a comment. Then provide your comments on the manuscript, data, and code if appropriate; it is helpful to authors if you separate these into “major” and “minor” points. Please avoid providing a very short report for a manuscript where you are recommending rejection unless the manuscript is completely unsuitable as it is important that the manuscript receives a fair and thorough review.
You may find this blog post useful to help structure your report.
What comments are appropriate as confidential comments to the Editor?
The Confidential Comments to the Editor section of the report form should be used to inform the Editor of any concerns about the ethics of animal or human research, the originality of the work, the integrity of the data presented or any potential undeclared conflicts of interest of the authors. This is also the section of the form where you should give your recommendation on whether the manuscript is publishable. Please don’t use this section to repeat your report, or include important information for the authors.
Can I edit my report after submitting it?
Unfortunately it is not possible for you to change your report once it has been submitted. Please contact the Editorial Office by email as soon as possible if you need to provide further information about the manuscript. However, while writing your report you can save a draft and return to edit that further, up until the point when you submit the report to the Editorial Office.
Revised manuscript stage
Will I be asked to review the manuscript again?
You may be asked to re-review the manuscript after the authors have completed revisions particularly if your report requested major revision. We always try to use the same reviewers to check revised manuscripts to avoid new criticisms being raised at this stage in the review process. You should avoid raising new points at re-review as it is unfair to authors if new points are raised at this stage unless these were not apparent in the first version of the manuscript.
What if the author has not made all changes I recommended?
The authors may have provided valid reasons for not making all the changes in the covering letter. However, if your assessment is that changes still need to be made, please outline this in your report together with reasons to help the Editor may make a decision. It is helpful if you alert the Editor to this in the Confidential Comments to the Editor section of your report.
What if the author has been rude?
In this situation we recommend that you stay professional and address the scientific comments without being drawn into an exchange with the author. Sometimes apparent rudeness is unintentional or the result of language difficulties. However, if you feel unable to continue with the review process, please contact the Editorial Office.
Why was a manuscript accepted when I recommended rejection (or vice versa)?
You may receive notice that a manuscript for which you recommended rejection has been published, or receive notification that it has been rejected when you recommended acceptance. This is not because your advice has been ignored. In making a decision to accept or reject a manuscript, Editors give full consideration to all reviewers’ reports and the authors’ response. If you see a published manuscript where the changes you requested have not been made in the final published version, this is likely to be because the Editor felt that the authors had adequately answered your criticisms.
How do I become a peer reviewer for your journal?
Please register with us on the publisher homepage and contact the Editorial Office of the journal for which you are interested in reviewing. Including a list of classifications and keywords for your subject expertise and interests will help us send you suitable manuscripts.
How long will I have to return my report?
We aim for quick manuscript turnaround time and a fast peer review process. You will be expected to submit your report within two weeks of accepting the invitation to review. The amount of time allowed for your review will be indicated in the invitation to review email. If you are unable to review within the time allocated, please let the Editorial Office know.
What if I submit my report after the deadline?
You will receive a series of reminders from the Editorial Office before your report is due. If you submit your report after the deadline, the handling Editor may choose not to use your report. We aim at quick manuscript turnaround time and fast peer review process and rely on reports being submitted on time.
What if the author contacts me directly?
Please contact the Editorial Office for advice on how to best proceed.
Can I contact the author directly?
Reviewers should not contact the authors directly as this can compromise the integrity of the peer review process. If you have an unusual situation where you would like to do this, please contact the Editorial Office first to explain why it would be useful.
What is portability of peer review?
To support efficient and thorough peer review, we aim to reduce the number of times a manuscript is re-reviewed after rejection from a BioMed Central journal, thereby speeding up the publication process and reducing the burden on peer reviewers. If a manuscript does not reach the interest criteria of a given BioMed Central journal, but is sound and in scope for another BioMed Central journal, we offer authors the option to transfer the manuscript together with the reviewer reports to the other journal.
Reviewers are asked whether they are willing to share their report with another BioMed Central journal in the event that the manuscript is rejected and transferred. This shares your name and report only with the Editor of the journal to enable them to make a decision regarding the manuscript’s suitability for the new journal.
If the Editor wishes to invite the authors to continue with their manuscript in the new journal, you may be invited to re-review the manuscript if further review is necessary. This speeds the review process, and decreases the burden on peer reviewers by reducing the need for new reviewers to assess the manuscript afresh. However, you are not obligated to do so and can decline the new invitation if you wish.
You can find more information on portability of peer review on our Editorial Polices pages.