Working with a third-party survey firm

Conducting a survey is a significant logistical undertaking. One decision research teams have to make is whether to outsource parts of the data collection process or run the survey in-house. The latter may still require hiring and training a field team (see also our resource on surveyor hiring and training). Research teams may decide to contract a survey firm, for instance, if the research organization has limited capacity or experience working with a given population, and there are established survey firms in the study area. However, survey firms may not have experience working on randomized controlled trials (RCTs) and may be more expensive. Contract negotiations can also be complex and time-consuming. This resource draws on the experience of J-PAL staff and researchers across regions on many different RCTs. The resource assumes the research team is designing, reviewing, and approving the study and the survey. This guide limits coverage of survey or study design to elements most critical to review with the survey firm. Questions related to survey design itself are covered in our Survey design resource. Examples are meant to be illustrative rather than prescriptive or comprehensive. 

The decision of whether to run a survey in-house or to outsource it depends on a number of factors: 

• Consider the capacity of the researcher’s organization to run a particular survey in-house within a given timeline. Survey firms may have existing infrastructure and networks of relevant stakeholders or vendors to handle complex or time-consuming processes, such as identifying and hiring the appropriate number and type of staff (both surveyors and supervisors), managing their contracts and payments, and handling logistics and field team management (organizing field visit plans, setting up transportation or accommodation, paying surveyors, etc.).
• Assess local knowledge, including experience working with the population of interest, languages spoken, modes and routes for transportation, and where to find accommodation. Consider whether existing survey firms have more or less knowledge and experience than the research organization in working with the intended population. You may also want to consider the firms’ experience with the study topic (e.g., education, health, or energy). If there are no survey companies with experience conducting projects in the area or with the population or topic of interest, consider hiring surveyors in-house. 
• Weigh the cost. Running a survey is expensive whether conducted in-house or through a firm. While it may be more expensive to hire a third-party firm, costs in this option may be more predictable since they are agreed upon ahead of time. Alternatively, a survey firm may appear to have lower costs but has not budgeted sufficiently for quality control. Consider the full cost in either case, including costs for the research team in reviewing proposals, negotiating contracts, conducting data quality checks, and whether some costs may be renegotiated. While specific budget estimates will be gained through the request for proposals (RFP) process, information about costs from previous engagements or general knowledge (sometimes specific firms are known to be particularly costly or inexpensive) can help you estimate the costs for a particular engagement. See the "Negotiating a Contract" section below for specific items that may be negotiated. 

If time permits, an open-bidding process can be useful for comparing firms on metrics such as cost and value-add. The different budget proposals can also provide an idea of which items can be negotiated to improve cost efficiency. If time is limited, single-source bidding (where a known or frequently used survey firm is solicited and then contracted) may be an option. Researchers should work closely with their organization’s procurement team to comply with any contracting requirements; universities and other organizations may require a competitive selection process for purchases or services priced over a certain threshold. They may also have specific requirements for the selection process, contracts, or terms. The process described below is typical of an open bidding process, though the timeline may vary based on the capacity of the team and the availability of survey firms. The five steps are explained in more detail below.

1. Identifying survey firms
2. Developing Terms of Reference (TOR)
3. Reviewing proposals from firms
4. Meeting with shortlisted firms
5. Negotiation and finalizing the contract 

1. Identifying survey firms

Speaking to local partners and other researchers who have worked in the study area, as well as conducting desk research can help identify a list of possible survey firms. For example, some J-PAL offices have a database detailing their previous work with survey firms. Consider the following factors when identifying and comparing potential survey firms:

• Research experience: Consider whether the firm has experience running a survey as part of an RCT or another type of impact evaluation. The expertise of high-level members such as advisors or directors (e.g., those with a research background) can add useful information.
• Survey methods and technology: Check if the firm has a specific set of skills, such as expertise in specific survey methods (e.g., mail, phone, in-person) or technologies that may improve the data collection process.
• Scale: Ask whether the firm successfully has conducted surveys at a similar scale and scope of the proposed survey.
• Organization structure: Make sure the firm structure fits the needs of a specific study. Complex surveys or those that require management by the research team may benefit from the structure of a smaller firm where employees are closely supervised. However, a large-scale study that covers a wide geographical area or has a substantial sample size may primarily require additional workforce, and large survey firms may have a stronger comparative advantage.  

After collecting and comparing information on an initial set of firms, the next step is to notify prospective firms. Organizations may have a particular system in place for these steps; if so, that process should be followed. The following section provides general information on what these steps may look like.

2. Developing Terms of Reference

A detailed description of the desired survey activities, study context, and aims of the overall evaluation is essential for potential survey firms to assess whether and how to bid for a contract. This description should be detailed in the Terms of Reference (TOR). The set of activities a firm will be hired to perform—whether a single round of data collection or all survey-related activities—are outlined in a subsection of this document: the Scope of Work (SOW).  While the SOW for which the firm will be contracted may cover only a small subset of overall evaluation activities, the TOR should provide enough context that the firm understands how the contracted work fits into the overall study. 

Specific formats and processes of developing a TOR may vary depending on the research project, organizational requirements, and local practices. A typical length is 5–10 pages, with appendices supplementing necessary information when appropriate. The TOR is usually attached to the legal contract.

Resources for writing TORs are available in the resources section below. See this generic TOR for a baseline survey, which includes sections on staffing, budgets, quality control, and timelines for deliverables, as an example. Major components that may be included in a TOR are outlined below.

1. An executive summary consists of a brief overview of the project, including the research objective and outcome(s) of interest. 
2. An introduction or contextual information section briefly orients firms to the history of the research project. The context may relate to the area where the data is collected or social, regulatory, or political factors that may influence the project implementation as it relates to contracted activities. 
3. An evaluation background and purpose frames the project, which helps firms understand the extent of the full evaluation. This can include the theory of change, the full timeline and geographic area of the study, and a summary of existing or planned activities related to the evaluation. Sending a survey firm grant applications for a project (without the budgets) can be a convenient way to provide background information.
4. A scope of work details the services the firm will be contracted to perform and the approach and methodology that the firm must take. This section includes key elements such as the sample size, process indicators to be measured, and availability of relevant data. Elements to include may vary with each project depending on the intervention, the process of data collection, and the activity to be subcontracted. This section can specify or request firms to propose any bonus or incentive schemes given to respondents and surveyors to ensure high-quality data and high response rates. In a competitive bidding process, assessing how firms interpret the approach and methodology of the data collection process is key to determining how much they understand. This section of firms’ proposals will allow bidders to differentiate themselves in terms of quality and how they plan to carry out the methodology proposed by the research team. Specifying the approach or providing an outline of activities can be the most challenging part of creating the SOW. A complete and detailed set of activities is not always possible, especially if contracting for the development of a process (i.e., a pilot survey), when it can be difficult to project a budget, timeline, and potential activities. In these cases, it is important to highlight that the SOW is for a pilot and that a certain amount of exploration is required. Emphasize what process or materials need to be developed within the SOW but include as much detail of known parameters as possible. For example, identify whether there is a fixed budget, set timeline, or sample size. 
5. Integrate quality assurance, including any procedures, such as checks and monitoring, that firms must perform to ensure high-quality data collection, as well as any required data security procedures. These steps may include requirements to: independently back translate (i.e., re-translate an already-translated questionnaire or other documents back into the original language) and pilot questionnaires before data collection begins; regularly monitor and audit the activities of surveyors; track the number of non-responses and have a strategy to address issues such as attrition; conduct double-data entry if using paper data collection; and regularly check error rates. Finally, this section can require the firm to conduct field monitoring in coordination with the research team or to allow time for members of the research team to review all training materials and attend trainings. Specific considerations with respect to quality assurance are detailed below in the "Ensuring a Successful Study" section.
6. A section outlining the key personnel and qualifications required to carry out the evaluation includes the size of the team needed, the required supervisor-to-surveyor ratio, a desired gender ratio (if applicable), the roles of each team member, the desired credentials or experience of each team member, and the role of team leader. Firms should submit curriculum vitae, as well as references or examples of past evaluation reports.
7. The timeline and deliverables should be detailed, explicit, and reasonable. Evaluate whether deliverables are feasible, given the project budget and timeline. In addition to the SOW, this section requires careful consideration, especially in the case of a pilot or other cases where ambiguity remains. Clearly discussing expectations in the TOR can prevent misunderstandings later in the survey process. Guidance on deliverables will generally include the following information: 
   1. Type of product: This can include real-time or periodic transfers of raw and/or processed data, codebooks, translated questionnaires, written study protocols, data security plans, training materials, implementation and field monitoring reports, reports on response rates, reports on quality assurance (e.g., a summary of high frequency checks), and products required by the project funders. Ideally, each product will be listed separately to specify each requirement and the timeline of when the firm should submit the product. 
   2. Structure and format of the product: If a written product, include the length of the report, the main content, and in what language it should be delivered. If a data product, include the format, file type, and documentation that should be delivered. Specify whether live access to data, raw data extracts, or formatted reports are necessary.
   3. Timeframe for product: This includes milestones with deliverables and timelines clearly listed (e.g., in bullet points or in a Gantt chart). 
   4. Required meetings or coordination: This establishes points of contact between the firm and the research team (e.g., weekly email updates, regular phone calls, or a written summary of insights from surveyors on lessons learned from conducting the survey). 
8. The budgets and payments can be open-ended or prescriptive. The TOR can state the overall budget and ask firms to propose what they are able to achieve within that budget. Alternatively, the TOR can specify budget items that should be included and ask applicants to budget according to these categories. The former may be useful in early stages of the research design, while the latter approach can provide a clearer comparison of each activity’s cost across companies. The budget section explains payment details, including the process for invoicing, the timeline, and the means of payment. Firms should be asked to submit a financial proposal with a spreadsheet or table of the proposed budget and a budget narrative that both justifies the cost and demonstrates that the budget is adequate to complete the required activities listed in the TOR. Requiring transparency in cost calculations, such as asking firms to specify unit costs of labor and materials, can be helpful in evaluating trade-offs between survey firms. It can be helpful to build flexibility in the contract, including in the budgets and payment section. For example, if contracting for a pilot, or a project where uncertainty (say around response rates) could alter methods substantially, one option is to ask for costs on a range of possibilities and negotiate on a menu of prices. After piloting different methods, or once uncertainty resolves itself (e.g., response rate becomes known), the research team is then able to choose the appropriate methodology based on a negotiated and known set of costs.

The TOR should be reviewed by relevant teams (e.g., the finance or contracts personnel at the research team’s institution) to ensure it is consistent with internal contracting and procurement procedures. Once the TOR is finalized and released, the applicant firm may wish to discuss and clarify the TOR.

3. Reviewing proposals from firms 

After the submission of proposals, two or three firms should be shortlisted, though this number can depend on a research institution's policy or the quality of submitted proposals. The shortlisting process is necessary to strategically decide the survey firms with which to meet for clarification and negotiations before signing a contract. To weigh the strengths and weaknesses of each company objectively, it is advisable to create a set of parameters based on predetermined criteria as described above. This process should include an initial budgetary review to seek which proposal gives the best value and is likely to deliver quality work. Creating a table that summarizes how each company’s proposal addresses the set of criteria will be helpful (see J-PAL's Survey Firm Assessment template for an example). It is also important to conduct reference checks either before shortlisting a firm or before finalizing the contract. 

4. Meeting with shortlisted firms

Meetings with shortlisted firms can be used to confirm whether they can deliver on the proposal and to discuss further details of the project. This may be a good time to discuss more ambiguous items in the TOR (e.g., if contracting for a pilot, to determine if the research team and firm have a similar understanding of the activities a pilot may include). In addition, ask questions about the submitted SOW and at what point a contract modification would be required. For example, confirm if the firm’s understanding of a pilot allows for changes in design, such as additional mailings, surveys, or phone calls.

These meetings can also be a good time to discuss details of the firm’s data collection and quality assurance procedures to understand whether they are able to deliver the quality needed for your survey. It is common practice to state that the initial meetings are part of the bidding process and that the team is still in the process of considering which firm to contract. 

5. Negotiating and finalizing the contract

Negotiating the budget

Negotiating the budget can be a long and complicated process that typically runs in parallel with the contracting process. The goals are to maintain a good relationship with the firm and get the project running within budget and without sacrificing quality. Before entering a negotiation meeting be sure to do the following: 

1. Assess whether the budget reasonably reflects all research activities, including surveyor salaries and pay for training days. If available, compare this budget to those from similar projects or budgets that the firm has submitted for previous projects. It is advisable to coordinate with someone with experience and local knowledge in this area, such as a local partner, a research manager, or a member of the finance team (if working with J-PAL or IPA).
2. Understand the budget limit and items that cannot be compromised. 
3. Note the items to negotiate. Based on our experience, key budget items that have the potential to be negotiated include the following:
   1. Questionnaire design (including programming and translation) may be done more cost-effectively by using resources available within the research team or organization. 
   2. Respondents' incentives (compensation for the time spent participating in the survey) depend on the type of respondents, the survey length, and the sample area. Survey firms may have experience that enables them to advise on suitable incentives for respondents (e.g., cash, vouchers, items). Consult the firm to identify an incentive that is within budget, logistically feasible to transport and distribute, and appropriately compensates the respondents for their time. See also MacKay’s blog post for a discussion on the ethics of payments to research participants. Buying in bulk may cut the costs of incentives.
   3. Institutional fees (may also be called overhead or management fee) can vary by firm. If the shortlisted firm has a higher fee and the overall cost is a concern, see if this is negotiable.
   4. Miscellaneous items should be evaluated to determine how essential they are for the project and whether they can be done in-house or in an alternate way (e.g., in-person vs. phone surveys).
   5. Confirm that the survey company has not missed any key activity in the data collection process (e.g., pilot, listing, conducting spot checks, and back checks). If there are any missing activities, consider revisiting the budget or SOW to ensure all key activities are included.

The negotiation process can be complex and delicate. As such, it should be led by senior-level staff, such as the principal investigators, project managers, or finance personnel. Staff leading the negotiation should come prepared with an understanding of the available budget, the flexibility in both the overall budget and specific line items, and how an overage would impact the rest of the project; additionally, staff should be aware of the necessity of maintaining good relations with the firm, particularly if it is the only viable candidate. 

Creating a contract

After the survey firm is selected, the research team should develop a contract in coordination with the relevant team within their organization (such as the finance team, procurement office, or the office of general counsel). Such teams may have template contracts available and will be familiar with legal provisions to ensure the research team is protected in case of breach of contract. 

The contract should address the rights and obligations of the parties involved. It should be specific and include the activities and requirements described in the TOR (often the TOR forms the basis for the contract or serves as an appendix), including any deliverables or requirements that have been further specified through the proposal and negotiation process.

Note that the contract is the main opportunity to formalize expectations and establish consequences for failure to deliver quality work. For example, consequences may include withholding future payments until quality issues are rectified or the ability to cut the contract short if necessary. As such, it is important to carefully draft and review the contract to be sure it allows for such contingencies. The more specific the contract, the easier it is for the survey firm to know what is expected and for it to be legally enforced or terminated. In addition to items from the TOR, the contract may include the following elements: 

• Financial reporting requirements
  • For example, this could include reports and projections of costs in a certain format following a specific timeline.  
• Quality checks and assurance
  • Detail the level of quality expected from data collected by the firm as well as the sorts of checks that should be performed to ensure that quality. It may also be helpful to include corrective actions to take if the data is not of the desired quality (e.g., re-surveying a portion of or the entirety of the sample) and be clear about the consequences of not delivering on this output (e.g., specifying that the survey firm must cover any additional re-surveying expenses that are the result of errors made by surveyors).
• Timeline and deliverables
  • In addition to the timeline and deliverables as described under Developing Terms of Reference, however, the contract should update the TOR as needed with specific (vs. relative) dates, activities, and deliverables, and specify payments or incentives associated with meeting (or failing to meet) deliverables. 
• Data ownership terms
  • The research team typically owns all data collected by a third-party survey firm, including respondents’ answers and contact information. 
• Confidentiality agreements
  • This restricts firms and their employees from sharing data and requires that personally identifying information (PII) be encrypted. Note that some firms may object to requirements for surveyors or their employees to sign confidentiality agreements (a requirement of any J-PAL project). In such instances, it is important to understand the processes the firm has for ensuring participants’ information is protected and to push back if necessary. See our resources on ethics and IRB for more.
• Contract amendment process
  • Amendments may be needed to modify the timeline due to delays or unforeseen conditions or to modify the SOW, such as in response to findings from a pilot or requests from the project partners or donors. 

After drafting a contract, expect to negotiate and modify the draft to meet the needs of both the researchers and the survey firm. Once the research team and survey firm agree on the contract, it should be reviewed by the finance, legal, or contracts team at the researcher’s organization and signed by someone with institutional signatory authority. 

When engaging a survey firm, the research team must decide which aspects of the survey they want to outsource. It is common to delegate to the firm survey logistics, recruitment, and field team management. However, it is important that the research team continues to manage or at least keep oversight of the surveyor training and data quality. It is especially crucial to be attentive to elements of the data collection that can affect the internal validity of study results. It is important to ensure that rigorous data quality checks are performed and that the survey is implemented consistently across all treatment groups.

The research team should design deliverables, milestones, and payment methods to ensure that incentives to collect high-quality data are aligned and can set expectations that enable real-time insight into project activities. Conducting surveys as part of an RCT also requires specific planning and coordination that may be new to otherwise highly experienced survey firms, and research teams should clearly communicate these RCT-specific considerations and data standards to the survey firm, as well as include those expectations within the contract. 

This section highlights topics specific to RCTs and study design, general study considerations, and project management recommendations that may be beneficial to review with the survey firm and for which to define clear responsibilities. Communicating about these topics in a way that acknowledges the survey firm’s expertise (the reason they were contracted) and identifies how certain activities or requirements feed into the overall study may allow for challenges to be identified early and for open conversations that can lead to productive resolution.

RCT-specific and study design considerations

Survey firms may have expertise in the mechanics of delivering surveys and identifying representative samples but do not necessarily have experience doing RCT-specific tasks, such as checking regularly for consistency in data collection procedures across treatment and control groups or delivering experimental modules (e.g., a module designed to observe differences in responses between treatment and control groups to a vignette). Work closely with the survey firm to ensure their expertise in survey implementation is complemented with these critical considerations for an RCT:

• The study intake, consent, and, if applicable, random assignment notification processes are designed to maximize response rates in the current and future rounds of the survey, maximize compliance with treatment assignments, and encourage accurate responses. The timing and implementation of these processes can have major implications for statistical power, bias, burden on respondents and/or surveyors, and cost. See the Define intake and consent process resource for more information. 
• Expected response rates will be adequate to achieve the necessary statistical power, and the respondents will be representative of the intended sample. Based on the experience of the survey firm and/or desk research, identify expected response rates for the full sample and any critical sub-groups. Work with the firm to explore ways to maximize response rates and allocate survey efforts appropriately. If balance along particular characteristics is important to the research design, ensure the survey contract specifies the composition of the sample, rather than just the overall size. See J-PAL's Sampling and Sample Size lecture slides and our Power calculations resource. You may also want to give specific instruction on replacement protocols for respondents whose data cannot be collected for particular reasons (e.g., respondents have permanently moved or are deceased). See our Survey logistics resource for more guidance on replacement for both in-person and remote surveys.

Although it is usually not advisable to delegate random assignment to the survey firm, in situations where it might be necessary (e.g., units are randomized on arrival or just after intake), protocols must be put in place to ensure that randomization is done correctly. If the survey firm is performing random assignment, emphasize that treatment assignment is not manipulable. The research team should also specify which randomization checks (e.g., running summary statistics on a set of predetermined characteristics) the firm will run and what output is needed to confirm randomization was done correctly. This is especially important if the research team will not have access to the randomized list or raw data. The Evaluating technology-based interventions resource provides additional guidance on how to check randomization procedure and sample when another entity is implementing the random assignment. In addition, the research team should perform independent randomization checks to ensure the randomization was done correctly.

Survey design and programming

Survey design

The research team should be responsible for designing, reviewing, and approving all questionnaires (see also our resource on survey design). However, the contract may specify the survey firm’s responsibility to translate the questionnaire into local language(s) and, as a quality assurance procedure, independently back translate the questionnaire (i.e., re-translate the translated questionnaire back into the original language, most often English.) 

Survey programming

Survey programming can be done by either the research team or the external firm, depending on available expertise with relevant survey softwares (e.g. SurveyCTO, SurveySolutions, ODK, etc.) and capacity. In either case, if the survey firm is responsible for translating and/or programming the questionnaire, the translated or programmed questionnaires should be included as deliverables with a clear timeline in the contract (see the section on timeline and deliverables below).

Surveyor hiring and training

Surveyor team structuring

The research team can set expectations about the size and structure of the team needed, including a required supervisor-surveyor ratio or a desired gender ratio (if applicable), as well as the desired credentials or experience of each team member. These expectations should be clearly defined in the Terms of Reference (TOR) that will later be added to the contract (see the section on Developing Terms of Reference above). 

If the research population comprises individuals who have experienced trauma or if the survey questions reference painful subjects, enumerators need to be equipped to interact with these individuals without inflicting further trauma and prepared to respond appropriately to signs of distress or requests for assistance related to the survey topics. In some cases, you may want to restrict the types of surveyors who will interview respondents if sensitive questions are involved (e.g., requiring that surveyors interviewing female respondents about intimate partner violence be women). Verify that the survey firm will identify surveyors with the appropriate skills, experience, and characteristics as outlined in the TOR. 

Surveyor training

It is difficult to overemphasize the importance of surveyor training. Training is a key opportunity for the research team to communicate directly with survey staff and make sure they understand the survey instruments, the survey protocols, and the related technology. Therefore, it is highly recommended to have research staff conduct or at least co-lead the training with the survey firm. Accordingly, the research team should develop or at the very least review training plans and protocols to make modifications as needed. See the Surveyor hiring and training resource for information on how to develop surveyor trainings as well as strategies for retraining, as necessary. 

In some cases, it may also be an option for the research team to conduct a training of trainers who would then lead the final surveyor training. In this case, the research team could attend the final training to ensure all relevant content is delivered and to answer questions. 

Depending on the IRBs the study is registered under, surveyors may also be required to undergo CITI training. If CITI training is not required for surveyors, alternative ethics training resources are available to ensure that surveyors receive some form of human subjects research training. This is covered further in the Ethical conduct of randomized evaluations resource. It is also important to include training on confidentiality, emphasizing the consequences of breaking participant confidentiality, in addition to having surveyors sign confidentiality agreements before starting fieldwork. See our Surveyor hiring and training resource for more guidance on data confidentiality agreements.

Survey logistics and field team management

Surveyor performance, pay, and incentives

Discuss how the survey firm tracks individual surveyor performance and decide whether the procedures are sufficient for the study. It is important to work with the firm to establish surveyor performance targets that are easy to assess, allowing for quick corrective action when necessary. Leveraging the firm’s knowledge of the local context, you can work together to determine a reasonable number of surveys to be completed by each surveyor based on the mode of survey delivery, survey length, the area in which the survey is being conducted, and the population being surveyed.

Depending on the firm, context, and project, researchers may have little to no control over surveyors and how they are paid. During the proposal review and interview stage, work to understand whether it is possible to design or adjust incentive structures to meet the study’s needs or to ensure established procedures do not pose a threat to data quality. Confirm the survey firm follows fair labor practices. See the Surveyor hiring and training and Field team management resources for more on surveyor performance evaluation, pay, and incentives.

Surveyor support

Depending on how the firm normally organizes its survey teams, having a reporting structure within a hierarchy of surveyor roles may be helpful when fielding multiple types of issues. Surveyors with more senior roles can be trained to support junior surveyors on specific matters (e.g., clarification on survey questions and protocols) while escalating more serious matters (e.g., concerns about safety) to management and the research team. Establishing a forum (e.g., a WhatsApp group) for surveyors to discuss difficult cases, troubleshoot, or ask questions amongst one another can also be helpful. Depending on the study’s aims and population, surveyors may be likely to experience distress or other emotional reactions. Support for surveyors is especially helpful in these cases. 

Data security and data quality checks

Data security measures

While a survey firm likely has established data security procedures, verify that all data security measures used, such as encrypting PII, using anti-virus software, and enabling remote data deletion capabilities, comply with your organization’s standards. Dealing with low-tech partners may also pose some challenges when it comes to ensuring data transfer and security. Non-disclosure and confidentiality agreements should be signed and enforced for any individuals accessing project PII. Reiterate the risks to participants of data breaches as well as the consequences of breaking these agreements (i.e., termination or suspension of work contracts). Data should also never be shared with any individuals who are not contracted by the survey firm. You should be cautious when working with firms who are reticent or refuse to sign these types of participant data protection agreements and can push back if a firm is unwilling to adequately describe its data protection processes, and if the processes seem insufficient. See the resource on surveyor hiring and training for more on confidentiality agreements and data security procedures for an overview of data security measures to use when working with data.

Data monitoring and management

In coordination with the firm, plan who will be responsible for data monitoring activities, and confirm how the firm will notify the research team and address any discrepancies, errors, or challenges identified. Confirm what data quality procedures the firm uses and identify whether additional actions are needed to ensure the resulting data will be suitable for the research study. 

Depending on the nature of the project and capabilities of the firm, research staff can conduct some elements of data monitoring themselves. Having research staff directly conduct spot checks allows the researchers to directly observe the survey or data collection procedures, thus developing deeper insight into the data and allowing for a more nuanced analysis. An additional benefit may be the ability to directly observe key aspects of the intervention or implementation aside from the survey, enabling a more thorough understanding of the study as a whole. See the Data quality checks resource for additional recommendations on data quality checks and monitoring.  

Data quality and cleaning

You should work with your survey firm to establish its role in ensuring data quality. Depending on what system they have in place for this activity, you may have to provide further instruction or technical support for checks that are more specific to your project. Include a requirement in the survey contract to share a data codebook or handbook, a description of the data storage plan, a description of data cleaning procedures, and other study protocols. It is strongly recommended that research teams regularly conduct independent checks to ensure that the correct data is being collected, that it is of high quality, and that it is not being falsified or manipulated in any way.

If possible, work with the survey firm to receive access to raw data. Sometimes a firm will be unable to share raw data or provide access to the data directly. In this case, written protocols and plans can facilitate researchers’ understanding of how the firm plans to clean and manage data in order to verify that procedures align with the goals of the research study. See more under the Data cleaning and management resource.

Project management recommendations

Using a third-party firm can make it more difficult to manage and monitor the study’s progress and budget. Below are management strategies to help ensure the project runs smoothly. 

Managing timeline

The survey(s) for which the firm is contracted may be only a subset of the overall project activities; for example, the research team or another entity may be responsible for implementing the intervention. To ensure the correct sequencing of survey and implementation activities, research teams may need real-time information on the survey activities, status, and timeline. The high-level timelines and due dates articulated in the TOR and/or contract likely will not include all interim steps required to complete all deliverables, and therefore may not be sufficient for use to track progress on the survey. The research team should consider asking for a detailed timeline or Gantt chart of activities and dependencies. Set expectations and create a plan with the firm to check in frequently on the timeline and to establish a system to notify the team if the firm anticipates delays. 

Managing budget

Being specific and thorough in creating an SOW at the RFP stage enables more accurate budget estimates. Nevertheless, project delays, project changes, or unforeseen circumstances can necessitate modifications to either the budget or the activities. Requiring detailed invoices or spending reports as a condition of the contract, reviewing those reports regularly (e.g., monthly), and discussing reasons for deviations from expected amounts, can allow the research team and survey firm to identify and mitigate challenges quickly. Mitigations may include adjusting the survey scope or methods or increasing the allowed budget. If there are indications the survey will go significantly over budget, refer to the financial terms in the contract and consider whether a contract modification is required. 

Last updated September 2023.

These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.