Resources for adapting to phone surveys during COVID-19

The safety of research participants and staff is of utmost importance to J-PAL. On March 17, J-PAL suspended all research activities that require in-person contact in order to limit the spread of COVID-19. Since the suspension of in-person research activities, J-PAL staff have worked to quickly adapt to phone and online surveys where possible to minimize disruption to ongoing research projects.

Over the past few months, we’ve developed new resources on transitioning to phone surveys, with guides that span from budgeting to remote trainings to quality assurance best practices. These resources, along with those developed by Innovations for Poverty Action (IPA), are listed below. They are intended to complement our crowd-sourced best practices by providing in-depth guides that address challenges specific to remote survey work. For tips on obtaining electronic signatures (e.g., for consent forms), see also J-PAL North America’s recent blog post. IPA’s RECOVR hub houses COVID-related questionnaires and will be updated as IPA’s technical resources are made public.

Planning the transition to CATI

• Transitioning to CATI checklist (12 pages) Saurabh Bhajibhakare, Ambika Chopra, Putul Gupta, and Mustufa Patel, J-PAL South Asia

  • A starting point for projects transitioning to phone surveys. Provides a high-level overview of the steps involved so that key tasks are not forgotten. The appendix includes detailed guidance on remote monitoring of surveyors, from hiring to tracking productivity and conducting weekly debrief sessions.
• Budgeting for phone surveys with remote monitoring (5 pages) Putul Gupta, J-PAL South Asia
  • This resource includes surveyor productivity estimates, a reference budget, sample team structures, and more. Please note that the costs and training scenarios are tailored to the J-PAL South Asia office and may need adapting to other contexts.
  • Key takeaways:
    • Budget for additional days of training even if the instruments are shorter and simpler, since projects will need to spend additional time training surveyors on protocols for conducting phone surveys (tracking calls and making appointments) as well as data security and transfer protocols. It is also important that each surveyor gets enough practice and attention, an element that becomes even more critical in the remote training setup.
    • Be conservative in productivity estimates, as a fair share of surveyors’ time is spent reaching out to potential respondents multiple times before success. For a 30 minute phone survey, we’ve found 4-8 surveys/person/day to be a realistic estimate; for shorter (10-15 minutes), we aim for 10-14 surveys/person/day.
    • Productivity estimate should take into account whether you’re trying to target a specific respondent (caregiver, household head) for the survey, which might mean more attempts to get that specific individual on the line.
    • Consider budgeting for one supervisor for 3-4 surveyors. The supervisor can listen in on calls, verify call logs, tally attempts and completed surveys at the end of each day, track surveyor start and end time, ensure appointments are being completed, etc.
    • Costs will change, particularly if surveyors are working from home. For example, travel allowances will not apply. If surveyors use their own devices, tablet rentals may not be needed, but surveyors will need number masking. Additionally, projects will need to budget for data packs for surveyors keeping in mind the nature of monitoring protocols (use of recordings) and training plan (platform, use of videos, etc.)

Training and quality assurance

• Remote trainings for field staff (7 pages) Putul Gupta, J-PAL South Asia
  • Synthesizing learnings from six J-PAL South Asia projects that launched phone surveys during the COVID-19 lockdown, this resource provides tips on planning for remote trainings, guidance on training effectively, and possible mitigation strategies to deal with challenges unique to remote training.
  • Key takeaways:
    • Do your homework on potential enumerators and ensure that they have the required equipment for phone surveys before they come to the training,
    • Choose your training platform carefully. Get your training participants to familiarize themselves with the platform ahead of the actual training.
    • Break the training plan into a number of small sessions to facilitate better comprehension and retention. Build in enough time into the training schedule for mock sessions and team debriefs. To overcome logistical hurdles, budget a few breaks of 10-15 minutes through the day when devices can be charged.
    • Train a small team of senior field staff ahead of the actual training and leverage their support in building resources and planning for the training.
    • Share sharp and concise resources that help training participants understand and apply what they learn during the training. Avoid sharing all the resources at once. Ensure that the training resources are appropriately formatted for viewing on mobile devices.
    • Leverage technology (videos, screen sharing, breakouts) to keep the participants engaged.
    • Utilize quizzes and demo recordings to identify knowledge gaps at the enumerator level as well as systematic gaps in understanding within the team for specific questions.
    • PRACTICE, PRACTICE, PRACTICE!
• Quality assurance for CATI (4 pages) Saurabh Bhajibhakare, J-PAL South Asia
  • This resource covers specific modifications to accompaniments, back checks, high frequency checks, audio audits, and daily supervision that can be made to accommodate phone surveys.
  • Key takeaways:
    • Closely monitor call status attempts, refusal rates (by time, day of the week), and the number of incorrect phone numbers in your sample from day 1. This information should actively feed into revising your productivity assumptions and budgets.
    • Accompaniments for phone surveys can be conducted through call conferencing (or a three-way call)—though note that IRB approvals are necessary if you plan to have a third party listen to calls for monitoring purposes. Be sure this information is included in the informed consent process. Staff might incur call or message costs and a reimbursement process should be put in place to cover this.
    • Audio recordings can function like accompaniments but done post hoc (though also require IRB approval). Options include SurveyCTO’s audio audit feature and call recording features from platforms such as Exotel. Recordings are enabled by default when using the web version of Exotel, but when using the Exotel field plug-in for SurveyCTO call recordings can be toggled on or off.
    • Call logs are a good place to check if the right phone numbers were called, the number of call attempts, and the duration of the calls. For SurveyCTO, use the early release versions of Android Collect 2.70.2+ to capture call logs using the phone-call-log() function. For Exotel, the call reports can be exported from the web platform.
    • Set up and test the SurveyCTO data flow for high frequency checks well in advance, including setting up import .do files and the code for running High Frequency Checks (HFCs). Check out IPA’s HFCs in Stata and J-PAL’s R code for HFCs. Also check out Stata utilities to work with SurveyCTO data (additional information on the sctoapi command here).

Technical solutions

• Need to conduct remote trainings in areas with limited internet access? IPA has developed a field plug-in to embed YouTube videos into SurveyCTO forms
  • More technical resources, not all COVID-specific, can also be found on IPA’s GitHub.
• If surveyors need to use their own devices, consider a call masking platform such as Exotel. R. Ramakrishna at J-PAL South Asia has developed a SurveyCTO field plug-in for Exotel calls and SMS.
• For step-by-step guidance on getting set up with Exotel, see the Quick Guide to Using Exotel (3 pages) Saurabh Bhajibhakare, J-PAL South Asia