This resource provides guidance on when to work with an external survey firm, the process of identifying and contracting a firm, and how to coordinate tasks to ensure a successful study. It highlights topics in survey or study design that may be beneficial to review with the survey firm, outlines a possible division of labor between the research team and the survey firm, and describes measures to take to ensure high-quality data collection when outsourcing (parts) of the data collection process. For complementary information on the data collection process, see also our resources on data quality checks, survey logistics, surveyor hiring and training, and field team management.
Conducting a survey is a significant logistical undertaking. One decision research teams have to make is whether to outsource parts of the data collection process or run the survey in-house. The latter may still require hiring and training a field team (see also our resource on surveyor hiring and training). Research teams may decide to contract a survey firm, for instance, if the research organization has limited capacity or experience working with a given population, and there are established survey firms in the study area. However, survey firms may not have experience working on randomized controlled trials (RCTs) and may be more expensive. Contract negotiations can also be complex and time-consuming. This resource draws on the experience of J-PAL staff and researchers across regions on many different RCTs. The resource assumes the research team is designing, reviewing, and approving the study and the survey. This guide limits coverage of survey or study design to elements most critical to review with the survey firm. Questions related to survey design itself are covered in our Survey design resource. Examples are meant to be illustrative rather than prescriptive or comprehensive.
The decision of whether to run a survey in-house or to outsource it depends on a number of factors:
If time permits, an open-bidding process can be useful for comparing firms on metrics such as cost and value-add. The different budget proposals can also provide an idea of which items can be negotiated to improve cost efficiency. If time is limited, single-source bidding (where a known or frequently used survey firm is solicited and then contracted) may be an option. Researchers should work closely with their organization’s procurement team to comply with any contracting requirements; universities and other organizations may require a competitive selection process for purchases or services priced over a certain threshold. They may also have specific requirements for the selection process, contracts, or terms. The process described below is typical of an open bidding process, though the timeline may vary based on the capacity of the team and the availability of survey firms. The five steps are explained in more detail below.
Speaking to local partners and other researchers who have worked in the study area, as well as conducting desk research can help identify a list of possible survey firms. For example, some J-PAL offices have a database detailing their previous work with survey firms. Consider the following factors when identifying and comparing potential survey firms:
After collecting and comparing information on an initial set of firms, the next step is to notify prospective firms. Organizations may have a particular system in place for these steps; if so, that process should be followed. The following section provides general information on what these steps may look like.
A detailed description of the desired survey activities, study context, and aims of the overall evaluation is essential for potential survey firms to assess whether and how to bid for a contract. This description should be detailed in the Terms of Reference (TOR). The set of activities a firm will be hired to perform—whether a single round of data collection or all survey-related activities—are outlined in a subsection of this document: the Scope of Work (SOW). While the SOW for which the firm will be contracted may cover only a small subset of overall evaluation activities, the TOR should provide enough context that the firm understands how the contracted work fits into the overall study.
Specific formats and processes of developing a TOR may vary depending on the research project, organizational requirements, and local practices. A typical length is 5–10 pages, with appendices supplementing necessary information when appropriate. The TOR is usually attached to the legal contract.
Resources for writing TORs are available in the resources section below. See this generic TOR for a baseline survey, which includes sections on staffing, budgets, quality control, and timelines for deliverables, as an example. Major components that may be included in a TOR are outlined below.
The TOR should be reviewed by relevant teams (e.g., the finance or contracts personnel at the research team’s institution) to ensure it is consistent with internal contracting and procurement procedures. Once the TOR is finalized and released, the applicant firm may wish to discuss and clarify the TOR.
After the submission of proposals, two or three firms should be shortlisted, though this number can depend on a research institution's policy or the quality of submitted proposals. The shortlisting process is necessary to strategically decide the survey firms with which to meet for clarification and negotiations before signing a contract. To weigh the strengths and weaknesses of each company objectively, it is advisable to create a set of parameters based on predetermined criteria as described above. This process should include an initial budgetary review to seek which proposal gives the best value and is likely to deliver quality work. Creating a table that summarizes how each company’s proposal addresses the set of criteria will be helpful (see J-PAL's Survey Firm Assessment template for an example). It is also important to conduct reference checks either before shortlisting a firm or before finalizing the contract.
Meetings with shortlisted firms can be used to confirm whether they can deliver on the proposal and to discuss further details of the project. This may be a good time to discuss more ambiguous items in the TOR (e.g., if contracting for a pilot, to determine if the research team and firm have a similar understanding of the activities a pilot may include). In addition, ask questions about the submitted SOW and at what point a contract modification would be required. For example, confirm if the firm’s understanding of a pilot allows for changes in design, such as additional mailings, surveys, or phone calls.
These meetings can also be a good time to discuss details of the firm’s data collection and quality assurance procedures to understand whether they are able to deliver the quality needed for your survey. It is common practice to state that the initial meetings are part of the bidding process and that the team is still in the process of considering which firm to contract.
Negotiating the budget can be a long and complicated process that typically runs in parallel with the contracting process. The goals are to maintain a good relationship with the firm and get the project running within budget and without sacrificing quality. Before entering a negotiation meeting be sure to do the following:
The negotiation process can be complex and delicate. As such, it should be led by senior-level staff, such as the principal investigators, project managers, or finance personnel. Staff leading the negotiation should come prepared with an understanding of the available budget, the flexibility in both the overall budget and specific line items, and how an overage would impact the rest of the project; additionally, staff should be aware of the necessity of maintaining good relations with the firm, particularly if it is the only viable candidate.
After the survey firm is selected, the research team should develop a contract in coordination with the relevant team within their organization (such as the finance team, procurement office, or the office of general counsel). Such teams may have template contracts available and will be familiar with legal provisions to ensure the research team is protected in case of breach of contract.
The contract should address the rights and obligations of the parties involved. It should be specific and include the activities and requirements described in the TOR (often the TOR forms the basis for the contract or serves as an appendix), including any deliverables or requirements that have been further specified through the proposal and negotiation process.
Note that the contract is the main opportunity to formalize expectations and establish consequences for failure to deliver quality work. For example, consequences may include withholding future payments until quality issues are rectified or the ability to cut the contract short if necessary. As such, it is important to carefully draft and review the contract to be sure it allows for such contingencies. The more specific the contract, the easier it is for the survey firm to know what is expected and for it to be legally enforced or terminated. In addition to items from the TOR, the contract may include the following elements:
After drafting a contract, expect to negotiate and modify the draft to meet the needs of both the researchers and the survey firm. Once the research team and survey firm agree on the contract, it should be reviewed by the finance, legal, or contracts team at the researcher’s organization and signed by someone with institutional signatory authority.
When engaging a survey firm, the research team must decide which aspects of the survey they want to outsource. It is common to delegate to the firm survey logistics, recruitment, and field team management. However, it is important that the research team continues to manage or at least keep oversight of the surveyor training and data quality. It is especially crucial to be attentive to elements of the data collection that can affect the internal validity of study results. It is important to ensure that rigorous data quality checks are performed and that the survey is implemented consistently across all treatment groups.
The research team should design deliverables, milestones, and payment methods to ensure that incentives to collect high-quality data are aligned and can set expectations that enable real-time insight into project activities. Conducting surveys as part of an RCT also requires specific planning and coordination that may be new to otherwise highly experienced survey firms, and research teams should clearly communicate these RCT-specific considerations and data standards to the survey firm, as well as include those expectations within the contract.
This section highlights topics specific to RCTs and study design, general study considerations, and project management recommendations that may be beneficial to review with the survey firm and for which to define clear responsibilities. Communicating about these topics in a way that acknowledges the survey firm’s expertise (the reason they were contracted) and identifies how certain activities or requirements feed into the overall study may allow for challenges to be identified early and for open conversations that can lead to productive resolution.
Survey firms may have expertise in the mechanics of delivering surveys and identifying representative samples but do not necessarily have experience doing RCT-specific tasks, such as checking regularly for consistency in data collection procedures across treatment and control groups or delivering experimental modules (e.g., a module designed to observe differences in responses between treatment and control groups to a vignette). Work closely with the survey firm to ensure their expertise in survey implementation is complemented with these critical considerations for an RCT:
Although it is usually not advisable to delegate random assignment to the survey firm, in situations where it might be necessary (e.g., units are randomized on arrival or just after intake), protocols must be put in place to ensure that randomization is done correctly. If the survey firm is performing random assignment, emphasize that treatment assignment is not manipulable. The research team should also specify which randomization checks (e.g., running summary statistics on a set of predetermined characteristics) the firm will run and what output is needed to confirm randomization was done correctly. This is especially important if the research team will not have access to the randomized list or raw data. The Evaluating technology-based interventions resource provides additional guidance on how to check randomization procedure and sample when another entity is implementing the random assignment. In addition, the research team should perform independent randomization checks to ensure the randomization was done correctly.
Survey design
The research team should be responsible for designing, reviewing, and approving all questionnaires (see also our resource on survey design). However, the contract may specify the survey firm’s responsibility to translate the questionnaire into local language(s) and, as a quality assurance procedure, independently back translate the questionnaire (i.e., re-translate the translated questionnaire back into the original language, most often English.)
Survey programming
Survey programming can be done by either the research team or the external firm, depending on available expertise with relevant survey softwares (e.g. SurveyCTO, SurveySolutions, ODK, etc.) and capacity. In either case, if the survey firm is responsible for translating and/or programming the questionnaire, the translated or programmed questionnaires should be included as deliverables with a clear timeline in the contract (see the section on timeline and deliverables below).
Surveyor team structuring
The research team can set expectations about the size and structure of the team needed, including a required supervisor-surveyor ratio or a desired gender ratio (if applicable), as well as the desired credentials or experience of each team member. These expectations should be clearly defined in the Terms of Reference (TOR) that will later be added to the contract (see the section on Developing Terms of Reference above).
If the research population comprises individuals who have experienced trauma or if the survey questions reference painful subjects, enumerators need to be equipped to interact with these individuals without inflicting further trauma and prepared to respond appropriately to signs of distress or requests for assistance related to the survey topics. In some cases, you may want to restrict the types of surveyors who will interview respondents if sensitive questions are involved (e.g., requiring that surveyors interviewing female respondents about intimate partner violence be women). Verify that the survey firm will identify surveyors with the appropriate skills, experience, and characteristics as outlined in the TOR.
Surveyor training
It is difficult to overemphasize the importance of surveyor training. Training is a key opportunity for the research team to communicate directly with survey staff and make sure they understand the survey instruments, the survey protocols, and the related technology. Therefore, it is highly recommended to have research staff conduct or at least co-lead the training with the survey firm. Accordingly, the research team should develop or at the very least review training plans and protocols to make modifications as needed. See the Surveyor hiring and training resource for information on how to develop surveyor trainings as well as strategies for retraining, as necessary.
In some cases, it may also be an option for the research team to conduct a training of trainers who would then lead the final surveyor training. In this case, the research team could attend the final training to ensure all relevant content is delivered and to answer questions.
Depending on the IRBs the study is registered under, surveyors may also be required to undergo CITI training. If CITI training is not required for surveyors, alternative ethics training resources are available to ensure that surveyors receive some form of human subjects research training. This is covered further in the Ethical conduct of randomized evaluations resource. It is also important to include training on confidentiality, emphasizing the consequences of breaking participant confidentiality, in addition to having surveyors sign confidentiality agreements before starting fieldwork. See our Surveyor hiring and training resource for more guidance on data confidentiality agreements.
Surveyor performance, pay, and incentives
Discuss how the survey firm tracks individual surveyor performance and decide whether the procedures are sufficient for the study. It is important to work with the firm to establish surveyor performance targets that are easy to assess, allowing for quick corrective action when necessary. Leveraging the firm’s knowledge of the local context, you can work together to determine a reasonable number of surveys to be completed by each surveyor based on the mode of survey delivery, survey length, the area in which the survey is being conducted, and the population being surveyed.
Depending on the firm, context, and project, researchers may have little to no control over surveyors and how they are paid. During the proposal review and interview stage, work to understand whether it is possible to design or adjust incentive structures to meet the study’s needs or to ensure established procedures do not pose a threat to data quality. Confirm the survey firm follows fair labor practices. See the Surveyor hiring and training and Field team management resources for more on surveyor performance evaluation, pay, and incentives.
Surveyor support
Depending on how the firm normally organizes its survey teams, having a reporting structure within a hierarchy of surveyor roles may be helpful when fielding multiple types of issues. Surveyors with more senior roles can be trained to support junior surveyors on specific matters (e.g., clarification on survey questions and protocols) while escalating more serious matters (e.g., concerns about safety) to management and the research team. Establishing a forum (e.g., a WhatsApp group) for surveyors to discuss difficult cases, troubleshoot, or ask questions amongst one another can also be helpful. Depending on the study’s aims and population, surveyors may be likely to experience distress or other emotional reactions. Support for surveyors is especially helpful in these cases.
Data security measures
While a survey firm likely has established data security procedures, verify that all data security measures used, such as encrypting PII, using anti-virus software, and enabling remote data deletion capabilities, comply with your organization’s standards. Dealing with low-tech partners may also pose some challenges when it comes to ensuring data transfer and security. Non-disclosure and confidentiality agreements should be signed and enforced for any individuals accessing project PII. Reiterate the risks to participants of data breaches as well as the consequences of breaking these agreements (i.e., termination or suspension of work contracts). Data should also never be shared with any individuals who are not contracted by the survey firm. You should be cautious when working with firms who are reticent or refuse to sign these types of participant data protection agreements and can push back if a firm is unwilling to adequately describe its data protection processes, and if the processes seem insufficient. See the resource on surveyor hiring and training for more on confidentiality agreements and data security procedures for an overview of data security measures to use when working with data.
Data monitoring and management
In coordination with the firm, plan who will be responsible for data monitoring activities, and confirm how the firm will notify the research team and address any discrepancies, errors, or challenges identified. Confirm what data quality procedures the firm uses and identify whether additional actions are needed to ensure the resulting data will be suitable for the research study.
Depending on the nature of the project and capabilities of the firm, research staff can conduct some elements of data monitoring themselves. Having research staff directly conduct spot checks allows the researchers to directly observe the survey or data collection procedures, thus developing deeper insight into the data and allowing for a more nuanced analysis. An additional benefit may be the ability to directly observe key aspects of the intervention or implementation aside from the survey, enabling a more thorough understanding of the study as a whole. See the Data quality checks resource for additional recommendations on data quality checks and monitoring.
Data quality and cleaning
You should work with your survey firm to establish its role in ensuring data quality. Depending on what system they have in place for this activity, you may have to provide further instruction or technical support for checks that are more specific to your project. Include a requirement in the survey contract to share a data codebook or handbook, a description of the data storage plan, a description of data cleaning procedures, and other study protocols. It is strongly recommended that research teams regularly conduct independent checks to ensure that the correct data is being collected, that it is of high quality, and that it is not being falsified or manipulated in any way.
If possible, work with the survey firm to receive access to raw data. Sometimes a firm will be unable to share raw data or provide access to the data directly. In this case, written protocols and plans can facilitate researchers’ understanding of how the firm plans to clean and manage data in order to verify that procedures align with the goals of the research study. See more under the Data cleaning and management resource.
Using a third-party firm can make it more difficult to manage and monitor the study’s progress and budget. Below are management strategies to help ensure the project runs smoothly.
Managing timeline
The survey(s) for which the firm is contracted may be only a subset of the overall project activities; for example, the research team or another entity may be responsible for implementing the intervention. To ensure the correct sequencing of survey and implementation activities, research teams may need real-time information on the survey activities, status, and timeline. The high-level timelines and due dates articulated in the TOR and/or contract likely will not include all interim steps required to complete all deliverables, and therefore may not be sufficient for use to track progress on the survey. The research team should consider asking for a detailed timeline or Gantt chart of activities and dependencies. Set expectations and create a plan with the firm to check in frequently on the timeline and to establish a system to notify the team if the firm anticipates delays.
Managing budget
Being specific and thorough in creating an SOW at the RFP stage enables more accurate budget estimates. Nevertheless, project delays, project changes, or unforeseen circumstances can necessitate modifications to either the budget or the activities. Requiring detailed invoices or spending reports as a condition of the contract, reviewing those reports regularly (e.g., monthly), and discussing reasons for deviations from expected amounts, can allow the research team and survey firm to identify and mitigate challenges quickly. Mitigations may include adjusting the survey scope or methods or increasing the allowed budget. If there are indications the survey will go significantly over budget, refer to the financial terms in the contract and consider whether a contract modification is required.
Last updated September 2023.
These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.
We are grateful to Eitan Paul, Jakob Gärtner, Kim Dadisman, Manasi Deshpande, Rebecca Dizon-Ross, Kim Gannon, Ray Kluender, and Neale Mahoney for their review and input. Theresa Lewis copyedited this work, and Amanda Lee formatted the guide, tables, and figures. Any errors are our own.
CITI Human Subjects Research (HSR), Social-Behavioral-Educational (SBE). The HSR trainings provide an overview and foundational training in human subjects research. It also covers “the historical development of human subject protections, ethical issues, and current regulatory and guidance information.” It should not be considered sufficient training in ethical concerns for researchers. The SBE course covers core human subjects research topics for social-behavioral-educational researchers. Additional modules are available that cover additional topics in more detail.
Generic Terms of Reference for a Baseline Survey. This document from the World Bank provides guidance on the preparation of a TOR for a baseline survey.
International Enumerators Human Subjects Research Training Guide. This page provides instructions for a paper-based human subjects training guide for international surveyors.
Survey Firm TOR resource. This resource from the World Bank’s Development Impact Evaluation (DIME) group provides an overview of a TOR for a survey firm, highlights the relationship with the SOW, discusses deliverables, and provides an example excerpt.
Theory of Change. This tool from the Annie E. Casey Foundation defines and guides readers on how to develop and use a theory of change model.