This resource guides researchers through background research and early discussions with a program implementer who has expressed interest in a randomized evaluation, and with whom a partnership seems potentially viable. It provides guidelines for researchers to conduct early conversations with twin goals: to build strong working relationships and to assess whether a randomized evaluation is feasible and appropriate for the partner’s context. It focuses on general guidance for researchers, but J-PAL specific resources for staff, affiliates, and invited researchers can be found at the bottom of this page.
Early discussions with partners serve dual purposes: (1) gathering enough information to assess the practical and statistical feasibility of a randomized evaluation, and (2) establishing strong working relationships with key stakeholders. By using initial conversations to discuss research questions, outcome measures, goals, and priorities, researchers and partners can jointly assess whether a randomized evaluation is in their best interests.1
Conducting independent due diligence on each potential project and implementing partner is critical to making an informed decision about 1) the feasibility of an RCT and 2) the compatibility of interests and motivations of both parties. Identifying the right partner can take time. Following the guidance in this document will help avoid pitfalls such as hastily selecting partners due to convenience, reputation or timing. Conducting due diligence often means exploring more opportunities so the best decision can be made for both the researcher and the implementing partner.
When entering a new thematic area or geographic area, strategies such as desk research, exploring professional networks and even attending relevant regional and national events that convene stakeholders can be a helpful place to start. Doing so will help researchers understand various actors (NGO, government, otherwise) working in specific fields of interest.
Tips:
After determining where and how to engage with potential partners, the next step is to prepare the discussion topics needed in order to assess the possibility of working together. Generally, in the “getting to know you” and “early scoping” phases of communications, information should be sought about the potential partner’s organization (history, funding, trajectory), programs (objectives, capacity, monitoring systems), and appreciation for learning and evaluation. This discussion will provide an indication of the potential partner’s willingness to learn about their programs, regardless of the outcome. For example, being prepared to use the same language as the partner (e.g., understanding whether a partner refers to the people they serve as patients or clients and using that term rather than participants), can help to foster trust and mutual respect.
Consider the following suggestions for these first conversations:2
With this groundwork, researchers and partners can begin to discuss what a randomized evaluation might look like in their context, keeping in mind that random assignment can take many forms7 depending on the research question and the needs of all stakeholders. Prospective partners may not have the research or technical background to understand benefits or costs of an evaluation design, nor which research design is most suitable for their goals.8 Researchers can provide information to support partners in making informed decisions about their participation in a randomized evaluation and the proposed study design.
If initial discussions with the potential partner seem promising, researchers can begin to think in more detail about potential research questions and design. In addition to literature review, background research may include continued discussions with the potential partner.9 These discussions, desk research, site visits, or focus groups can illuminate:
Equipped with this information, researchers and partners can assess the fit of a randomized evaluation. If the partnership seems viable, researchers can use this information to tailor their approach to communicating with the partner, to offer trainings or resources as appropriate, and to design a rigorous and practical randomized evaluation.
Last updated March 2021.
These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.
We are grateful to Todd Hall, Emma Rackstraw, and Sophie Shank for their insight and advice. Jacob Binder copyedited this document. This work was made possible by support from the Alfred P. Sloan Foundation and Arnold Ventures. Any errors are our own.
“Evaluating Financial Products and Services in the US: A Toolkit for Running Randomized Controlled Trials.” 2015. Innovations for Poverty Action. November 10, 2015. https://www.poverty-action.org/publication/evaluating-financial-products-and-services-us-toolkit-running-randomized-controlled.
Page 18 of IPA’s Evaluating Financial Products and Services in the US: A Toolkit for Running Randomized Control Trials provides more questions determine whether a randomized evaluation is right in a particular context.
Glennerster, Rachel. 2017. “Chapter 5 - The Practicalities of Running Randomized Evaluations: Partnerships, Measurement, Ethics, and Transparency.” In Handbook of Economic Field Experiments, edited by Abhijit Vinayak Banerjee and Esther Duflo, 1:175–243. Handbook of Field Experiments. North-Holland.
For more information about whether a randomized evaluation is right for a given program or partner, see Section 1.2 of Rachel Glennerster’s The Practicalities of Running Randomized Evaluations: Partnerships, Measurement, Ethics, and Transparency (available as a part of a Handbook of Economic Field Experiments, published at Elsevier).
“What is the Risk of an Underpowered Randomized Evaluation?” J-PAL North America.
This resource outlines why an underpowered randomized evaluation may consume substantial time and monetary resources while providing little useful information.
Gugerty, Mary Kay, and Dean Karlan. n.d. “Ten Reasons Not to Measure Impact—and What to Do Instead.” Stanford Social Innovation Review. Accessed October 2, 2018.
In their book The Goldilocks Challenge, and summary article in the Stanford Social Innovation Review, Mary Kay Gugerty and Dean Karlan discuss the appropriateness of randomized evaluations in different situations and alternative strategies where appropriate.
Glennerster, Rachel, and Shawn Powers. 2016. “Balancing Risk and Benefit: Ethical Tradeoffs in Running Randomized Evaluations.” The Oxford Handbook of Professional Economic Ethics, April. https://doi.org/10.1093/oxfordhb/9780199766635.013.017.
See Rachel Glennerster and Shawn Powers’s Balancing Risk and Benefit: Ethical Tradeoffs in Running Randomized Evaluations for a detailed discussion about ethics in randomized evaluations.
"Common Questions and Concerns about Randomized Evaluations." J-PAL North America.
It is helpful to review this resource, along with Why Randomize and Real-World Challenges to Randomization and Their Solutions (both listed below), before initial conversations with a potential partner in order to be reminded and informed of examples and ways to explain RCTs in non-technical language. These resources can also be sent to a potential partner after the first meeting or call with a partner so as to clarify or expand on concepts of RCTs.
“Real-World Challenges to Randomization and Their Solutions.” J-PAL North America.
Includes more information about solutions to practical challenges to randomization, as well as case studies.
J-PAL video resource: Why Randomize?
This resource is helpful if the potential partner wants to learn more about monitoring and evaluation.
Feasibility Checklist (Hoekman 2019) (J-PAL internal resource)
J-PAL’s guide to early-stage engagement with partners (J-PAL internal resource)