Assessing viability and building relationships

Early discussions with partners serve dual purposes: (1) gathering enough information to assess the practical and statistical feasibility of a randomized evaluation, and (2) establishing strong working relationships with key stakeholders. By using initial conversations to discuss research questions, outcome measures, goals, and priorities, researchers and partners can jointly assess whether a randomized evaluation is in their best interests.¹ 

Conducting independent due diligence on each potential project and implementing partner is critical to making an informed decision about 1) the feasibility of an RCT and 2) the compatibility of interests and motivations of both parties. Identifying the right partner can take time. Following the guidance in this document will help avoid pitfalls such as hastily selecting partners due to convenience, reputation or timing. Conducting due diligence often means exploring more opportunities so the best decision can be made for both the researcher and the implementing partner. 
 
When entering a new thematic area or geographic area, strategies such as desk research, exploring professional networks and even attending relevant regional and national events that convene stakeholders can be a helpful place to start. Doing so will help researchers understand various actors (NGO, government, otherwise) working in specific fields of interest.  
 
Tips:

• Access research databases of NGOs and government programs in the country or region of interest; take note of interesting and relevant programs.
• Look for RFPs and funding opportunities (such as the World Bank) that often have matchmaking components for researchers and implementing partners.
• Leverage networks and common connections for introductions, but don’t hesitate to make new connections. Be mindful that, while cold-calls or cold-email approaches can be successful, they will likely require more time and commitment to initiate the conversation.
• Stay up to date on upcoming events where practitioners and researchers meet, especially those that attract practitioners more inclined to participate in a rigorous impact evaluation. For example,  J-PAL’s event website, IPA’s event website, 3IE’s event website, and CGDev’s event website are all useful resources to find relevant events.
• When considering a partnership with a government, read Researchers Should Collaborate With Governments for Both Sides to Be More Effective.

After determining where and how to engage with potential partners, the next step is to prepare the discussion topics needed in order to assess the possibility of working together. Generally, in the “getting to know you” and “early scoping” phases of communications, information should be sought about the potential partner’s organization (history, funding, trajectory), programs (objectives, capacity, monitoring systems), and appreciation for learning and evaluation. This discussion will provide an indication of the potential partner’s willingness to learn about their programs, regardless of the outcome. For example, being prepared to use the same language as the partner (e.g., understanding whether a partner refers to the people they serve as patients or clients and using that term rather than participants), can help to foster trust and mutual respect.

Consider the following suggestions for these first conversations:²

• Generate trust by starting conversations from the partner’s needs and constraints, rather than from the “ideal” research design. Ask key stakeholders, program managers, and frontline service providers about their program, what they hope to learn, and what challenges they face. Such conversations will provide information needed to refine the research question and gather information that pertains to study feasibility and design. For example, discussions about the number of individuals served over a certain timeline can serve dual goals of informing power calculations and better understanding the program and partners. 
• Be sure the partner’s goals for an academic partnership align with what researchers can promise. For example, if a partner is hoping to “prove” that some aspect of their program works, clarify that you cannot guarantee positive results. If a partner places a high priority on an underpowered outcome, be sure to explain the dangers of running an underpowered evaluation.³ From this conversation, gauge how the partner might use or react to the results from an evaluation—whether those results are positive, negative, neutral, or inconclusive—and be sure they understand your intention to publish results regardless of the outcome. Note that there may be fruitful cases where researchers’ and partners’ goals are not exactly the same, but for which the marginal cost of meeting both is very small. However, identifying a design that is both feasible and addresses the partners’ and researchers’ goals may be an iterative process. 
• Explain the basic concept of a randomized evaluation, potential opportunities for random assignment, and related studies or random assignment possibilities. This conversation will help to assess the potential partner’s willingness to incorporate random assignment into their operations and identify potential challenges.⁴
• Inquire about time or resource constraints the potential partner may be facing. Ensure that a potential partner understands the length of the research process and likely duration of their involvement. If the potential partner has never participated in academic research before, the timeline may be much longer than they expect. Furthermore, the financial runway for both research and program resources should be factored into the matchmaking decision.
  • Note: In most cases, at least 3-4 months is required for researcher and implementing partner matchmaking, project development and piloting. The baseline survey may require several more months.
• Establish an equal partnership. Demonstrating a willingness to design research that is appropriate to the context and minimally disruptive to the partner’s operations, while also demonstrating respect for the partner’s insight and contextual knowledge, may alleviate concerns about the effect of the research on the community and partner.⁵ This view of partnership is important for the full study team to internalize and return to throughout the study.
• Identify a champion within the organization. This may be the initial point of contact or someone with a particular interest or background in research. For example, a senior-level official who is intrinsically motivated to understand whether and how their program works can assist in developing a high-level vision for the partnership, implementing the partnership, facilitating support at all levels (including lateral buy-in), and helping ensure project sustainability (Carter et al. 2018).⁶

With this groundwork, researchers and partners can begin to discuss what a randomized evaluation might look like in their context, keeping in mind that random assignment can take many forms⁷ depending on the research question and the needs of all stakeholders. Prospective partners may not have the research or technical background to understand benefits or costs of an evaluation design, nor which research design is most suitable for their goals.⁸ Researchers can provide information to support partners in making informed decisions about their participation in a randomized evaluation and the proposed study design.

If initial discussions with the potential partner seem promising, researchers can begin to think in more detail about potential research questions and design. In addition to literature review, background research may include continued discussions with the potential partner.⁹ These discussions, desk research, site visits, or focus groups can illuminate: 

• How the program operates in practice. This includes whether services are delivered with fidelity to the design, the extent of variation (formal and informal) in program implementation, the history of changes to the program, and potential future changes. This also illuminates where random assignment could plausibly and ethically be carried out, which people can be recruited and consented into the study, and what potential threats to the research design exist. 
• Program and partner details. This includes growth trajectory, sample size, time frame in which people are served, retention rates, and data capabilities. Gathering information on relevant programmatic facts within the first few conversations is an encouraging sign of a partner’s interest in collaborating on an evaluation and of their ability to provide relevant data. These details can inform whether a partner is likely to have the technical and logistical capacity to implement the program, reach enrollment targets, and provide data needed for an evaluation. 
• Key personnel and other stakeholders. These may be within the organization or external parties such as funders or government partners. Gather information on their roles, involvement in decision-making, familiarity with research, and relation to one another.
• A timeline of major changes that may affect the program, such as:
  • A similar program that may be implemented in the same service area
  • Changes to data systems or processes
  • Changes in program funding
• Potential implications for the community or individuals from study implementation, including:
  • The relevance of the issue or program to the population involved
  • Ethical standards¹⁰ and any community or population-specific concerns¹¹ (e.g., young people, people involved in the justice system, or pregnant women)
• Representativeness of the community context or study population. Communities or partners willing to adopt a program for a randomized evaluation may respond differently to the program than those who are not (Allcott 2015). 
• Potential sources of concern about research or a randomized evaluation. Understanding the community’s relationships with universities, research, and researchers—particularly any other ongoing research in the area, any perception of high fatigue from prior surveys, or any history of research that the community perceived as harmful—can help researchers plan for potential challenges or concerns from partners. Concerns may especially involve groups whose circumstances have made them vulnerable to manipulation or from pre-existing controversy over a program or political decision.

Equipped with this information, researchers and partners can assess the fit of a randomized evaluation. If the partnership seems viable, researchers can use this information to tailor their approach to communicating with the partner, to offer trainings or resources as appropriate, and to design a rigorous and practical randomized evaluation.

Last updated March 2021.

These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.