Sharing experiences of measuring women’s and girls’ empowerment in impact evaluations

Celebrating International Women’s Day at J-PAL Southeast Asia through a workshop on J-PAL’s Practical Guide to Measuring Women’s and Girls’ Empowerment

Policymakers and practitioners around the world are rallying around Sustainable Development Goal (SDG) 5: to achieve equality and empowerment for all women and girls. Recent movements to end gender inequality and sexual harassment demonstrate that all countries need to work towards rectifying harmful power imbalances based on gender. To make progress towards this vision, we need a better understanding of effective policies to empower women and girls, and we need to know how to measure empowerment well.

As a part of J-PAL’s global research network, J-PAL Southeast Asia (SEA) is among the many organizations working to identify programs to reduce gender inequality and grappling with how to best measure women’s empowerment through impact evaluations.

On March 12, J-PAL SEA and members of J-PAL’s Gender sector team hosted a workshop for over 40 policymakers, NGO program staff, donor representatives, and development practitioners in Jakarta to share experiences and lessons learned in designing, monitoring, and evaluating programs to increase women’s agency and empowerment. A majority of the participants had been involved in measuring the results of their organizations’ programs and half came from organizations that are currently working on women’s empowerment interventions.

Drawing on diverse disciplines and the experience of J-PAL affiliated researchers around the world, the interactive workshop focused on tools researchers and practitioners can use to apply a gender lens in program and M&E design. This included discussions on how to design survey questionnaires to measure women’s empowerment, whether using standardized questions or developing your own tools.

“We are pleased to see J-PAL SEA's progress in enhancing achievements in gender equality and women's empowerment in all of its activities, both in research, policy outreach, and capacity building of researchers in Indonesia.” – Lucia Pietropaoli (Counsellor, Poverty and Social Development, Australian DFAT)

Tailoring survey questions to fit into the local context and designing data collection strategies to respect respondents’ time can often be challenging. During the workshop, Glory Sunarto, from Mercy Corps Indonesia, reflected on her experience collaborating with J-PAL affiliated professors, the Center for Global Development, the World Bank, and a large public bank in Indonesia on a joint evaluation measuring the impact of the bank’s mobile banking program on supporting financial inclusion of female entrepreneurs.

An important takeaway from her experience was to streamline the information that needs to be gathered in the data collection process, especially in a multi-partnership evaluation. While each organization involved may have different outcomes they wish to measure, they may not all translate well into survey questions that will be easily understood by the respondents. Furthermore, respondents can become fatigued during long surveys and their answers may become less accurate as time passes.

Speakers and participants also highlighted the importance of considering how gender intersects with other aspects of identity like ethnicity, class, and disability in program and M&E design, and analyzing how multiple forms of oppression overlap and further magnify inequalities. When asked what they plan to take away from the workshop, around 53 percent of participants noted that they plan to use the information shared to guide their research and 60 percent plan to leverage the knowledge gained to inform their program implementation.

A participant shares his ideas on gender transformative programs targeted to change men's views on societal norms. Photo: Alifia Firliani | J-PAL 

Uncovering gender dynamics and measuring empowerment in impact evaluations can be challenging. During the workshop, Claire Walsh (Senior Policy Manager, J-PAL Global) shared several tips from J-PAL’s guide:

1. Good measurement depends on understanding gender dynamics and what women want in a particular context. Conducting formative research to gain a deeper understanding of these things is an important step in designing and evaluating any program. For example, it can help us understand how women’s choices are constrained in a particular context, generate a theory of change about how a program could work, and identify what language to use in survey questions.
2. Think about complementing standardized measures with locally-tailored ones. Standardized questions are useful for comparing outcomes across interventions and places, but they may not always be relevant to or work in a particular context. Complementing standardized questions with locally-tailored ones may allow us to capture subtler changes in the local context where we’re working.
3. Pilot data collection instruments before starting data collection. Testing our instruments with real people from the communities in which our evaluation will take place is critical for understanding whether we are asking the right questions, if the questions make sense to the people we are interviewing, and whether they are picking up the information we hope they will.
4. Gender dynamics are also present in the data collection process. Survey participants may feel less free to speak openly about certain topics with enumerators from a particular gender, class, and/or ethnicity. Piloting offers a good opportunity to investigate how enumerator identity may influence the data collection process. For sensitive topics, participants may value ways to answer survey questions privately, for example, by using audio computer-assisted self-interview technologies or interactive voice response calls.

To read more on developing a measurement strategy and data collection tools that are appropriate to your context and purpose, learn tips for overcoming common challenges in measuring empowerment, and see catalogues of survey modules and non-survey instruments used in previous evaluations, see J-PAL’s Practical Guide to Measuring Women’s and Girls Empowerment in Impact Evaluation.