Best practices for conducting phone surveys

Posted on:
Authors:
Sarah Kopper
Anja Sautmann
blue mobile phone
Photo: David Batcheck | IPA

Thank you to everyone who attended the webinar with Tavneet Suri and the UBI Kenya team on Tuesday, March 24. We have posted a recording of the webinar, as well as Tavneet's slides, SurveyCTO code, and the modified questionnaire.

The protection of staff and respondents who participate in our research is of highest priority for J-PAL. As of March 17, J-PAL has therefore suspended all research activities that contribute to in-person interactions in order to limit COVID-19 spread. Many other research organizations have done so or will do so soon. In order to disseminate expertise quickly and build on our combined networks, we are crowd-sourcing best practices on switching from in-person to surveying online or via phone. 

This is a living document that aggregates crowd-sourced tips and factors to consider when conducting remote surveys (in particular phone surveys) while practicing social distancing. We will be updating this page regularly. Many thanks to those who have contributed to this collection thus far.

Do you have experience conducting surveys online or via phone? You can contribute via this form. 

Last updated: April 14, 2020

Four resources that cover many of the topics below are:

Resources from Tavneet Suri's webinar on phone survey use:

Other references (not reviewed):

  • Check with the IRB of record about moving to verbal consent. If you have approval for written consent, you will need to submit an amendment. If you already have approval for (in-person) verbal consent, you may not need to submit an amendment. 
  • Keep the informed consent script short and use simple, clear language. Be sure it includes the purpose of the call, who is calling (organization and individual), confidentiality, and duration of the survey. Pilot it internally over the phone, to get a sense of the length and whether it is easily understood. You may need to make several revisions before implementation.
  • The survey protocol should log when verbal consent was given. The survey should not continue unless consent was explicitly confirmed by the enumerator. Consent should be given by clearly speaking a short phrase, such as “Yes, I agree”. (Poppy Widyasari, J-PAL Southeast Asia)

See also sample survey protocols for the UBI Kenya project (Tessie Lezcano, Eunice Kioko, and Debborah Muthoki, IPA)

  • The risk of data loss might increase when enumerators survey from home. Is there a higher risk that devices are lost or stolen? Are enumerators able to upload data to a secure server (and off of their devices) in real-time? Depending on the answers to these questions, you may need to update the discussion of risks and benefits in an amended IRB application. (Ben Morse, J-PAL Global)

Finding the respondent:

  • Collect multiple contact details for each respondent to maximize response rates over rounds. Herath et al. 2019 compiled tips from running phone surveys in urban centers of South Africa. They collected participants’ current number(s), a family member’s phone number, and a friend’s phone number. Because these details are so important, they recommend putting in place multiple measures to avoid data-entry errors (e.g., double-entry, length constraints, etc.). They also collected email addresses, but found that email wasn’t an effective channel for communication. 
  • You may also collect different contact channels. For example, a survey in Indonesia collected respondents’ WhatsApp number, Line, Facebook, WeChat, IMO, or Skype ID, and found that most respondents preferred WhatsApp or Skype calls.

Making sure the respondent picks up:

  • Sending a text message ahead of time improves response rates. Kasy & Sautmann (2020) in collaboration with PAD conducted a test of how to maximize response rates for phone enrollment to an extension service (see experimental results in this dashboard). Response rates varied between 13 and 20% (“cold-calling” farmers from a government curated list of phone numbers). The most successful was a morning call at 10am, preceded by a text message an hour in advance. Morse et al. (2016) found that sending text messages about 5 minutes before the first call attempt was helpful. The best time of day and the best interval between text message and call may vary, depending on the population. 
  • Use social media as an alternative to SMS to contact the respondents and explain the purpose of the call before calling (provided there is IRB approval/consent for these forms of contact) (Yuna Liang, IPA).
    • Depending on the size of the survey, radio programs are an alternative means of informing respondents about the study (Sarah Hughes, Mathematica)
  • Set up protocols for failed call attempts by making the call at different times of the day (e.g., in 3-hour intervals). This holds whether the calls are made via automated systems or by surveyors. Determine a number of attempts that must be made over several days before considering the respondent unreachable. (Tessie Lezcano, IPA and Grant Bridgman, Uliza).
    • The protocols should include a system for recording when the first attempt was made, including the time of day and day of the week. Tracking the timing of successful calls can tell you when future calls are most likely to be successful. Be sure to pre-test the system. (Sarah Hughes, Mathematica)
    • If response rates remain low, increase the number of attempts (e.g., from 9 to 12). (Tessie Lezcano, IPA)
  • In Indonesia, a 37% response rate was achieved by contacting respondents three times to set up an appointment and then calling at the appointed time. In Turkey, a 75% response rate was achieved just by placing more calls at different times of the day, with a large boost coming from placing calls in off-hours. (Ozler & Cuevas 2019)
  • Build overtime into the study to be able to call participants outside of their work hours. This helps to reach those who cannot speak on the phone during the day, and it ensures that the study isn’t biased by only capturing findings of specific groups. (Herath et al. 2019).
  • Brand your phone number if you can, and build trust and recognition around your brand (put up posters, mention it on the radio, tell node members of the community) (Grant Bridgman, Uliza).
  • In some contexts, obtaining and sending out an official letter, e.g., from the government, before survey start can increase response rates.
  • Run a de-duplication on the phone numbers in your sample so that respondents aren’t called multiple times (Sarah Hughes, Mathematica)

Sometimes the unexpected works:

  • Respondents may pick up when a new number calls. Let enumerators switch their call lists of unreached respondents; the respondent may pick up a new number. (Yuna Liang, IPA).

Completing the survey:

  • Delivery matters: enumerators should sound professional. A polished script can keep respondents on the line. (Grady Killeen, Precision Agriculture for Development (PAD)/HBS)
  • Build on established rapport between enumerator and subject. Morse et al. (2016) used phone surveys to track trends in health service usage during the Ebola crisis in Liberia. They had the advantage of surveying respondents in person at baseline, using phone surveys for follow-up. When possible, the same enumerator was assigned for both baseline and follow-up. 
  • Positive, unambiguous interactions are helpful for re-establishing contact. Be clear with participants at the outset on when and how communication will take place over time, and be open about how long a survey is likely to take. Also explain why the survey is being conducted over the phone, particularly if earlier contact was in person. (Herath et al. 2019 and Michael Kleinman, Orange Door Research).
  • Compensating respondents via mobile money or airtime for the time spent on the survey and the costs incurred from having to charge phones makes participation easier and more attractive for them.
    • In a survey in Ghana, we paid 3 cedis (about $1.30 US) per call and had 85% completion rate (and if a call was missed, the vast majority of the time, the person answered the next call: complete dropout was rare.)  (Rachel Heath, University of Washington).
    • If airtime can’t be sent electronically, surveyors can purchase airtime refill cards and read the code off to respondents after the survey is complete. (Ben Morse, J-PAL Global)
  • Including the duration of the survey in the consent procedure will reduce instances where the respondent runs out of time or phone battery and cuts the interview short.
  • Build in as many options as possible to do the survey at the respondent's convenience. For example, rescheduling to another time or day, calling on another phone number, or allowing the respondent to interrupt to take incoming calls.
    • Be sure that a refusal is in fact a real refusal, rather than a respondent who just doesn't have time right then. (Tessie Lezcano, IPA)
  • If you are going to be conducting phone surveys with the same respondents over time, ask them in the first survey what time is appropriate to call them during weekdays or weekends. This is particularly important for rural households who go to the field, have poor network connections, share phones or charge their phones irregularly. This kind of information can also help you plan your survey better when working on a different project in the same area or region. Knowing your targeted respondents patterns of life and phone usability is key. (Simon Rubangakene, PAD)
  • With participants with access to smart phones, email, and/or computers: use a combination of electronic notifications and provide multiple options for response (e.g. email, text messages, app pushes for notifications, and for responses, telephone call-in to recorded question/answer, link to website, texting back and forth, emailing, app questionnaire). Use telephone calls only for non-response follow up. Make the number of questions, and the questions themselves, short; if need be, survey frequently (e.g., every three weeks).  Compensate for completed surveys immediately. We have used these methods in the U.S. and have achieved response rates of over 85 percent (Jim Greiner, Harvard Law School).
  • Know particular contexts and the usability of SMS or online services. For example, in some places, SMS is not widely used and so may not be an effective means of communication. (Simon Rubangakene, PAD)
  • Surveyors may have trouble finding quiet places to conduct surveys. Consider buying them noise cancelling headphones, altering working hours to take place during quiet times, and helping them set up a room to avoid sound disturbances (e.g., by filling it with soft furniture). (Sarah Hughes, Mathematica)
  • Surveyors may need equipment such as a 4G adapter, dongle, or SIM card. If possible, try to deliver to them directly so they do not have to go to the market to make purchases. Put enough money on the SIM card for the survey (or set up a system for sending surveyors phone credit), but strongly enforce the message that the SIM card is for the survey only. (Sarah Hughes, Mathematica)
  • SurveyCTO is developing a starter kit to facilitate the transition from CAPI to CATI using the SurveyCTO Collect mobile app and will have a software update that includes a dialer plug-in and the ability to create field plug-ins at the end of March. More COVID-related resources and updates from SurveyCTO can be found here. (Chris Robert, SurveyCTO)
  • Twilio is a good platform for sending SMS and can also be incorporated into your customized application (available countries here). (recommended by several people)
  • Other commercial options include Telerivet (worldwide), Vonage APIs (formerly Nexmo), Africa’s Talking, Geopoll (CATI and SMS), Viamo (IVR), engageSPARK, and Survey Solutions (Grant Bridgman, Uliza; Sarah Hughes, Mathematica; and Michael Kleinman, Orange Door Research)

  • Uliza.org specializes in putting together interactive voice response (IVR) solutions in local languages. These are automated calls that allow respondents to answer in different ways. Uliza has also developed voice-based ID software using unique voice characteristics. This is useful when you have repeated engagements with someone over the phone to verify that you are talking to the person who gave the initial consent. Uliza also provides a (human mediated) service that allows researchers to receive open-ended voice questions/comments from their sample in local languages, and give answers that will be transmitted back in local-language voice (Grant Bridgman, Uliza).

  • If an internet connection is available, VOIP may be an option. VOIPStudio can be configured so that calls are coming from a real number, not an unidentified number. (Eduardo Vargas Sanchez, J-PAL LAC)

  • 60 Decibels specializes in designing and implementing large-scale voice, SMS and IVR surveys for the social sector, focused primarily on sub-Saharan Africa, South Asia and East Asia. 60 Decibels partners with aid organizations, social enterprises, impact investing funds, corporations, foundations, civil society organizations and donors. (Sasha Dichter and Ashley Speyer, 60 Decibels)
  • See pages 6-8 of the Remote Survey Toolkit from 60 Decibels for pros and cons of different technology solutions, as well as a list of remote survey providers. (Sasha Dichter and Ashley Speyer, 60 Decibels)
  • Though currently at capacity, Orange Door Research specializes in designing and implementing large-scale CATI, IVR, and SMS surveys for the development and humanitarian sectors, focused primarily on otherwise difficult-to-access geographies. Orange Door Research partners with UN Agencies, the World Bank, academics, civil society organizations, and donors. (Michael Kleinman, Orange Door Research)
  • Try working with local service providers: In Benin, we are working directly with the mobile service provider MTN to use StandardPro, a service that will permit recording the calls and that can help us with managing the phone dialing in a distributed fashion (Sarah Hughes, Mathematica)
  • Keep phone surveys short: at most 30 minutes. Figure out your primary outcomes and be sure those are included. You may need to aggregate questions, including some outcomes. If your survey is especially long, consider breaking it up into multiple phone calls. (Tavneet Suri, MIT)
  • Eliminate questions that are sensitive as far as possible (for example on stress, mental health, sex, domestic violence, etc) as these are harder to administer over the phone. Simplify or eliminate questions that require a lot of probing.
    • If you do need to ask sensitive questions and the respondent is surrounded by other people, ask if they can be called back later when they are alone. (Tessie Lezcano, IPA)
  • For new surveys: Before starting the main survey, conduct a check on a small sample of phone numbers to see how many are unreachable/ wrong numbers/ etc. This will give you an upper bound on the possible response rate, and you can adjust your sample size accordingly.
  • For new surveys, you may also need to adjust for lower response rates; 50% may be a good rule of thumb, though response are generally higher in the places we work than in the US. (Michael Kleinman, Orange Door Research)
  • Respondents may be charged for incoming phone calls in some locations. Consider a protocol that sends them some airtime credit before calling. 
  • Created drop down menus for question responses where possible; even if they are long, doing so can reduce entry errors and reduce the time enumerators spend typing (Grady Killeen, PAD/HBS and Michael Kleinman, Orange Door Research)
  • Random digit dialing and asking the village chief/elder for phone numbers are two options for sampling when the research team doesn't have access to a comprehensive list of phone numbers. Random digit dialing is just what it sounds like: set the relevant prefixes (e.g., country code), then randomly generate numbers to call. (Tavneet Suri, MIT)
  • The Busara Center has tips for adapting lab-in-the-field protocols to be done via phone; many of these tips also apply to field projects done outside the lab. Topics include specific suggestions for changing protocols based on the target sample and study protocols, as well as alternatives to focus group discussions. (Busara Center via Wim Louw, J-PAL Africa)
  • In countries where many languages are spoken, track languages on the tracking sheet so that respondents can be reassigned to a surveyor who speaks the language. (Tessie Lezcano, IPA)
  • See also sample survey protocols for the UBI Kenya project (Tessie Lezcano, Eunice Kioko, and Debborah Muthoki, IPA)
  • Though we are not qualified to provide medical advice, it may be useful for surveyors to have information about the nearest health clinics on hand, or to remind respondents about the importance of hand-washing and social distancing during the conversation (Tavneet Suri, MIT Sloan; Subha Mani, Fordham; Bidisha Barooah, 3ie)

Hiring

  • Ensure fluency in the local languages represented in your sample. (Sarah Hughes, Mathematica)

  • Include in the scope of work and surveyor contract that the phone numbers for your sample may not be used for any other purpose than your survey. (Sarah Hughes, Mathematica)

Training

  • Find a way to conduct small group calls, so one enumerator can conduct the interview with a pilot respondent, and other enumerators can listen in. After the interview, discuss what went well and what could be improved. Don't forget to ask respondents’ consent to be on the phone with several enumerators. (Yuna Liang, IPA)
  • Record a test interview to listen to as an alternative to a group call.
  • You may need a higher trainer to trainee ratio and more time than previously budgeted for if you are doing a remotely based training (Grant Bridgman, Uliza).
  • Use mock scripts (not just role-play) for training and testing each interviewer's ability to dial the correct number, gain cooperation, administer each question correctly and enter each response. Create a confidentiality agreement that each interviewer must sign. (Sarah Hughes, Mathematica)
  • Depending on the size of the team and at-home internet access, training of new enumerators can be done remotely via Skype, Zoom, or similar (Tavneet Suri, MIT Sloan)

See also Herath et al.’s (2019) blog post for general tips on training and practicing.

Keeping data quality high:

  • Increase the frequency of back-checks and HFCs, as well as the amount of time the field manager and RA spend looking through data, e.g., at least 50% of the field manager’s time daily. One target for back checks is to do them on 20-25% of respondents (up from 10-15%). (Tessie Lezcano, IPA and Sarah Hughes, Mathematica)
    • Pay extra attention to filter questions (skip logic): questions for which answering "no" skips a module or set of questions. (Sarah Hughes, Mathematica)
  • Use back checks to confirm that consent was given. (Grady Killeen, PAD/HBS)
  • If it is hard to find back check questions that should not have changed from the original survey, take a different approach. Ask questions such as, "Was the interviewer polite?", "Were you offered an incentive?", and "How long did the survey take?" (Sarah Hughes, Mathematica)
    • Check the respondent's reported duration against the timestamped data.
  • In the endline survey, include questions that should not have changed from the baseline survey. (Tavneet Suri, MIT)
  • Supervising field staff should call each field team member daily for a quick check-in and to go through any questions they might have, as well as go over their tracking sheet. (Tessie Lezcano, IPA)
  • You may consider recording calls for quality control and to conduct spot checks as long you have IRB approval and the subject gives their consent. Another surveyor can fill the survey questionnaire from the audio recording in order to conduct data analysis (e.g. determine error rates) on the back checks.
  • Set up a Whatsapp or Facebook chat group for each enumerator team where they can share issues or ask questions, and report each successful interview (this is particularly motivating if the supervisor gives positive feedback and encouragement, or even disburses small rewards such as phone credit). Research team members should be in the group as well to answer any questions. (Tessie Lezcano, IPA and Yuna Liang, IPA)
  • Use the SurveyCTO function of exporting data to google sheet to make a live monitoring tool of survey progress. Set the SurveyCTO app to auto sending completed forms, and use the google sheet monitoring dashboard to auto-update with incoming data. (Yuna Liang, IPA)
  • Looking at call metadata can help spot unusual patterns, such as an enumerator making very short calls. (Rachel Heath, University of Washington).
    • SurveyCTO also has a sensor metadata function that can capture the volume level of the environment where the SurveyCTO form was filled. While there is no set standard of “reasonable values,” outliers on the lower end (e.g., 10 decibels) could flag surveys that require additional scrutiny. (Yuna Liang, IPA)
    • Add the start and end time to SurveyCTO to see call duration. Call logs can also be used to see if field staff are making calls during the correct timeframe. (Tessie Lezcano, IPA)
  • Use the same team that you trust whenever possible but also bring in new enumerators to check patterns in the data for recurring surveys. (Simon Rubangakene, PAD)

Evidence on data quality

Garlick et al. (2019) compared weekly in-person, and weekly phone surveys for a 12-week microenterprise survey panel. They report: “The results show few differences across the groups in measured means, distributions, and deviations of measured data from an objective data-quality standard provided by Benford’s Law. However, phone interviews generated higher within-enterprise variation through time in several variables and may be more sensitive to social desirability bias.”

Webinar Recording: Adaptions for phone surveys with Tavneet Suri

Authored By

Research Methods

Previous

Next