This section covers the essentials of survey design. It includes an overview of survey development, practical tips, formatting suggestions, and guides to translation and quality control.
There are five key steps in the process of designing effective questionnaires:
This page focuses on Step 4, questionnaire design. As the diagram above suggests, it is important to be clear on your research questions, theory of change, and concepts to measure before starting to design the questionnaire. See the Introduction to measurement guide for resources on Steps 1-3 and Questionnaire piloting for resources on Step 5.
As described further in Introduction to measurement and indicators, indicators are used to measure outcomes or concepts. The early stages of survey development involve gathering ideas and beginning to formulate questions through:
If you are designing a follow-up survey, be sure to use the previous round’s questionnaire as your starting point. To facilitate comparability across survey rounds, only modify questions when absolutely necessary. In general, it is better to remove or add questions than to change them.
Even when designing a new instrument, you should not start from scratch but instead start by reviewing existing, well-tested surveys that occur in the same country or sector. When compiling questions from existing surveys, be sure to add a column noting the source of each question. Good sources of survey questions include:
At the highest level, questions may relate to facts or to perceptions/subjective expectations, but within these categories, you may also need to gather data that is observational, sensitive, or that relates to things that people do not know well. This section details key considerations and practical advice for each of these types of questions.
When measuring facts:
Subjective questions assess subjective psychological states and are not verifiable by external observation or records. They aim to measure a number of different things:
Practical advice for subjective questions:
What makes a question sensitive depends on culture and context, but information relating to identity, illegal activities, and socially unacceptable behavior are almost always sensitive. Respondents may not answer truthfully due to social desirability bias or embarrassment, or because they feel that a different answer is strategic. The following strategies can be used to deal with sensitive questions:
See the World Bank DIME wiki for more information on sensitive questions (J-PAL staff and affiliates: see also this lecture from J-PAL South Asia's Measurement & Survey Design course).
Respondents may have difficulty answering questions on anything they have to estimate, particularly when they are estimating across time. They may be prone to error due to forgetfulness, computational errors or weak numeracy skills. There are strategies you can use to minimize errors:
See also the World Bank DIME Resources lecture on Survey Instruments.
Observational questions, in which the enumerator infers something about the subject from observing them rather than direct questioning, mitigate against respondent bias and can be effective in monitoring the program (see the Implementation monitoring resource), verifying the existence of household assets or items of facility infrastructure, and more.
One possible disadvantage, particularly when assessing behavior, is the possible existence of Hawthorne effects. That is, subjects may alter their behavior due to their awareness of the interviewer.
Tips for observational questions:
Each question should be SMART:
There are a number of different ways of formulating questions:
Designing good response codes requires background and qualitative research. Ensure that responses are:
Response scales can be used to gather information, though all may be subject to the central tendency bias (i.e., a tendency to choose the middle option). Types of response scales include:
In broad terms, questionnaires are generally structured as follows:
Wherever possible, questions should flow as follows:
Ensure that you use skip patterns, particularly when surveys are long. Skip patterns direct the flow of questions so that respondents answer only those questions that are relevant to them.
Mistakes with skip patterns can lead to two possible types of errors:
Skip patterns can easily be coded into your surveys using software such as SurveyCTO.
You may also want to consider including the following as part of your survey materials:
Most surveys are now digital.1 If you need to use paper surveys, you should bear in mind the following best practices:
In some scenarios, face-to-face interviews may be infeasible. While the principles behind phone survey design remain the same as for in-person interviews, here are a few additional crowd-sourced considerations in a blog post compiled by Kopper and Sautmann (2020) to make when adapting an existing instrument to phone surveying:
For more practical guidance on phone survey design, see J-PAL South Asia’s checklist for transitioning a survey to CATI and the phone survey protocols from the UBI Kenya project. For a larger overview of phone survey considerations and collection of resources, see J-PAL’s Best practices for conducting phone surveys blog post.
Translating the survey into the mother language of your respondents speeds up the survey process and minimizes the risk of in-field translation errors. Best practices for translating are:
If translating a survey, it is particularly important to:
Do not underestimate the time needed to get a good translation, and be sure to have a professional translator—fluency in both languages is not a sufficient condition of a good translation. If translation is done by field staff, rather than a professional translator, it is useful for several staff to work on the translation together to reach a consensus on the most appropriate and likely to be understood translation for particular words and concepts. For technical surveys, it is useful to find someone who also has some subject knowledge.
You should also have the survey available in English. Most survey software programs allow for multiple languages, with users selecting the appropriate language at the start of each survey. See also the DIME Wiki’s page on translation.
Once you have a draft of the survey, in both English and any local languages, you should:
Ensuring high-quality control during primary data collection is an ongoing task that requires continuous effort. For further information on this, refer to the different types of data quality checks.
It is vital to document changes to surveys, or the way questions should be interpreted, and to communicate this to relevant parties. Be sure to choose a survey system that is easy to update (SurveyCTO, for example, has good version control), and provide full instructions for using and interpreting the instrument. See more in Survey programming.
Last updated September 2023.
These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.
We thank Serene Ho, Mike Gibson, and Jack Cavanagh for helpful comments. All errors are our own.