Pre-results review at the Journal of Development Economics: Taking transparency in the discipline to the next level

Earlier this year, the Journal of Development Economics (JDE), with support from the Berkeley Initiative for Transparency in the Social Sciences (BITSS), began the pilot of a “pre-results review” track for academic authors who submit their work to the journal for peer review and publication.

Pre-results review means that the study is reviewed before there are any empirical results. In Stage 1, authors submit a proposal that includes an introduction, methods, and data analysis plan for a prospective study. High-quality studies are accepted based on pre-results review, which constitutes a commitment by the JDE to publish the upcoming paper, regardless of the results. Authors can then collect and analyze the data and submit a full paper for final review and publication (Stage 2). Stage 2 review makes sure that the final paper is aligned with the research design accepted in Stage 1.

J-PAL interviewed JDE Editor in Chief Andrew Foster, Co-editor Dean Karlan, and BITSS Faculty Director Edward Miguel, to get some early insights from the pilot thus far.

How did you decide that the field of development economics was ready for pre-results review?

We hope that it is—the best way to learn is to try!

In development economics, we have long argued for the use of rigorous evidence to inform decisions about public policy. As a result, development economists are increasingly asking questions that are causal in nature and conducting experiments to answer them. This methodological turn also applies to economics as a whole.

However, high-quality studies that deal with important questions but fail to show theoretically tidy or statistically significant results, often go unpublished, or worse, they are not even written up! As a result, we are left with an evidence base of papers that tell ‘neat’ and clean stories but may not accurately represent the world. We believe that journals must publish well-designed and well-executed studies, even if their results do not allow for easy, clear-cut interpretation.

Philosophy aside, this pilot seemed like an innovative way to bring more good research to the Journal. On the side of the JDE, what got the ball rolling was that our publisher transitioned to a manuscript handling platform that enabled a two-stage peer review process; that we had a supportive co-editor in Dean who was willing to take on a chunk of the editorial work; and that Ted and BITSS secured the necessary resources to support the pilot through staffing, research, and the development of editorial materials.

What do you see as the benefits of the pre-results review model?

We tell our graduate students: “If a question is worth asking, then it shouldn’t matter what the result is.” We’ve tried to put this idea into practice as editors at the JDE.

But long before a final paper is submitted for peer review, there is ample room for good work to be left on the cutting-room floor because of the perception that some results are “less publishable.” By reviewing papers before the results are in, we hope to eliminate such fears and empower authors to present all results confidently and transparently. Note that we don’t intend to downplay the importance and value of exploratory data analysis that was not described in a pre-specified analysis plan: such analysis can be included in a paper, as long as they are clearly denoted as such.

We also see benefits for scholars who are entering the academic job market or are in the tenure review process. Given that field projects in development economics may sometimes take years to complete, acceptance based on pre-results review can be an early signal of the quality of a prospective or ongoing research project. Once accepted based on pre-results review, we offer authors to have the title and the abstract of their project posted publicly as a prospective article on the BITSS website.

Because peer review in this track takes place before fieldwork, this model provides authors with the opportunity to fine-tune their research design early in the project’s lifecycle. We envision pre-results review as a platform for constructive dialogue between authors and referees, focused on improving the accuracy and feasibility of already high-quality research designs.

If accepted based on pre-results review at the JDE, can authors still try to first publish the paper at another top economics journal, such as the American Economic Review?

Yes. In fact, we encourage this.

Our guess is that a good fraction of the papers that will be accepted based on pre-results review will, in fact, appear in the JDE. However, we also consider it a success if they end up in other top journals of the discipline, as this helps establish the merit of the pre-results review process.

Following acceptance based on pre-results review, authors are welcome to submit their full manuscripts to another journal, so long as they notify us and acknowledge in their published paper that their work was first reviewed and accepted by the JDE through this track. A manuscript will only be withdrawn from JDE if and when it is accepted by the other journal.

How do you hope this new model will impact the profession? How will you know whether the pilot was a success?

We hope that it opens up a venue for innovative high-quality research projects that otherwise may not happen due to concerns about the difficulty of publishing certain types of results. A successful pilot will result in new, high-quality JDE papers that are at least as impactful as those filed through the regular submission track.

We will consider it an even bigger success if, after being accepted based on pre-results review, some papers end up in other top journals and enrich the discipline with insights and results that may have otherwise gone unpublished.

In what ways do you think pre-results review could lead to fundamental changes in the reviewing process over the long run?

A number of things could happen.

For example, authors could use pre-acceptance by the JDE to obtain funding for their studies. We imagine that the biggest effect may be on junior researchers. If pre-results review helps researchers with novel and workable ideas get the recognition they deserve, this is certainly a positive outcome.

But we also recognize that, as with any review process, we will not always make the right call. That is why other options are available to authors, including sending a regular submission to the JDE or another journal at a later date.

Do you see any risks to the JDE as the first journal to adopt this model in the discipline?

First, we are mindful of the fact that a two-stage review process puts an extra burden on our co-editors and reviewers. We are extremely appreciative that our reviewers have been very helpful and supportive. In our view, the discipline’s move towards greater transparency and empiricism in the last few years may have helped ease the transition into this new model.

Second, even though we did a fair bit of background research on the experience of journals in other disciplines that have used this model, we continue to discover issues that seem especially salient for development economics.

For example, at journals in other disciplines, authors have normally found it relatively straightforward to implement their research according to the designs approved at Stage 1. But based on the feedback we have received from authors and our own experience in the field, we anticipate that there will be studies in this pilot that cannot be completed as planned for reasons beyond the control of the researcher, including objective circumstances such as political instability or sudden policy changes.

While minor adjustments are possible as long as they are transparently discussed, in some cases it simply may not be possible to follow through on the proposed analysis at all. Regardless of the outcome, we will make sure to draw the necessary lessons for the future.

Do you expect other economics journals to follow suit anytime soon? What is your advice to other journal editors?

It is hard to know how things will evolve, but we are fairly optimistic that other economics journals will introduce pre-results review in the future. For example, the new American Economics Review: Insights journal could be a natural venue for pre-results review, given their willingness to publish alternative article formats.

Our advice for journal editors is to proceed with a flexible mindset and to make sure to record and facilitate the learning process, both from the perspective of the journal and the authors.

This is a cross-post with the CEGA blog. The questions for this interview were developed with support from Anja Sautmann (Director of Research, Education, and Training, J-PAL).