Power calculations

• The sensitivity of an experiment to detect differences between the treatment and the control groups is measured by statistical power.  
• A type I error is a false positive: falsely rejecting the null hypothesis of no effect, or falsely concluding that the intervention had an effect when it did not. The probability of committing type I error is known as α.
• A type II error is a false negative: failing to detect an effect when there is one. The probability of committing type II error is typically given by β. To differentiate it from the treatment effect β, in this resource we denote type II errors by κ.
• Power is the probability of rejecting a false null hypothesis. Formally, power is typically given by 1-β. Again, to differentiate power from the treatment effect β, in this resource we will denote power by 1-κ. That is, maximizing statistical power is to minimize the likelihood of committing a type II error.
• Power calculations involve either determining the sample size needed to detect the minimum detectable effect (MDE) given other parameters, or determining the effect size that can be detected given a set sample size and other parameters. 

Components of power calculations include:

• Significance (α): The probability of committing a type I error. It is typically set at 5%, i.e., α=0.05.
• Power (1-κ): Typically set at 0.8, meaning the probability of falsely failing to reject the null hypothesis is 0.2 or 20%.  Power mirrors the significance level, α: as α increases (e.g., from 1% to 5%), the probability of rejecting the null hypothesis increases, which translates to a more powerful test.
• Minimum detectable effect (MDE): The smallest effect that, if true, has (1-β)% chance of producing an estimate that is statistically significant at the α% level (Bloom 1995). In other words, the MDE is the effect size below which we may not be able to distinguish that the effect is different from zero, even if it is. 
• Sample size (N)
• Variance of outcome variable (σ²)
• Treatment allocation (P), the proportion of the sample assigned to the treatment group. Power is typically maximized with an equal split between treatment arms, though there are instances when an unequal split may be preferred.
• Intra-cluster correlation coefficient (ICC): A measure of the correlation between observations within the same cluster, also often given as ρ.  If the study involves clustered randomization (i.e., when each unit of randomization contains multiple units of observation), you will need to account for the fact that individuals (or households, etc.) within a group such as a town/village/school are more similar to each other than those in different groups.¹ In general, this will increase the required sample size.

Power is also affected by the design of the evaluation, the take up of the treatment, and the attrition rate, discussed in greater detail below. 

Hypothesis tests focus on the probability of a type I error, or falsely rejecting the null hypothesis of β=0, i.e., falsely concluding that there is a treatment effect when one does not exist. As written above, significance levels are typically set at 5%, but in taking this conservative approach we increase the risk of a different error: failing to detect a treatment effect when one exists, i.e., a type II error.² Power is defined as the probability of NOT committing a type II error. That is, power is correctly rejecting the null hypothesis when a treatment effect exists, or [1-prob(type II error)].

 The graph below depicts how a type II error could happen:

Power and sample size equations

There are two main approaches to calculating power, depending on whether the study’s sample size is fixed:

• If the sample size is fixed according to budget constraints or external factors (e.g., the number of eligible children in a partner’s schools), power calculations determine the effect size the study is powered to detect (the MDE). Researchers then decide whether the study should go forward as designed or whether the design should be tweaked to increase power. Formally³:

• If the sample size is not fixed or capped at a predetermined number, power calculations can be performed to determine the minimum sample size needed to detect an effect size (the MDE) given the parameters:

Relationship between power and power components

The figures and equations presented above provide additional intuition for the relationship between power and its components, holding all else equal. This section discusses this intuition then summarizes the relationships in a table below.

Outcome variance and sample size: Sample outcome variance decreases as sample size increases, as larger samples are more representative of the underlying population. This leads to a more precise treatment effect estimate, which in turn lowers the probability of a type II error and hence increases power. The same is true of outcomes with less variation in the underlying population because sample variance will typically be smaller when the population variance is smaller. Note that this can also be seen analytically using the calculation of standard errors above: as the sample size (N) increases, or as the population variance (𝜎²) decreases, the standard error will decrease.

Figure 2 shows that as the variance of the beta decreases, the null and the alternate hypothesis become narrower, a larger portion of the alternate hypothesis lies to the right of the critical value (critical value represented by the dashed yellow line) and the power increases (teal shading).

This section provides an overview of sources for each component of power calculations and tips for running initial calculations. It also covers guidance on refining calculations based on interest in binary or multiple outcomes and design features such as unequal cluster size or stratified randomization.

Gathering inputs

Some components of power calculations are based on decisions or assumptions made by the researcher and partner, while others are calculated using existing data.

Based on decisions or assumptions:

• Power (1-κ) is typically set at 80%, or 0.8, though in some cases it is instead set at 90%. While it can be tempting to increase a study’s power, doing so also increases the risk of a false positive (type I error). See Cohen (1988) for more information. 

• Significance level (α) is typically set at 5%, i.e., α=0.05.
• Units of randomization and observation: The units of randomization and observation do not need to be the same. See the “Implementation” section of the Randomization resource for more information on determining each. In clustered designs, while the addition of more clusters will in general increase power more than the addition of more units to existing clusters, the former is typically more expensive to do.
• MDE may be a practically relevant effect size such as the minimum improvement in outcomes required for the implementing partner to determine the program’s benefits are worth the costs or to scale the program. Note that published studies evaluating similar interventions will typically provide an optimistic estimate of a reasonable effect size. This difference will be diluted by any non-compliance, so the expected effect size should be adjusted to the expected compliance rate in the two groups.
  • If information about realistic effect sizes is not available, Cohen (1988) proposes rules of thumb that rely on standard deviations⁴: 0.2 standard deviations for a small MDE, 0.5 standard deviations for a medium MDE, and 0.8 standard deviations for a large MDE. These standards are based on social psychology lab experiments among undergraduates, however, and have since been argued not to be realistic in real-world settings for some sectors and regions--see e.g., Kraft (2019) (associated blog post) and Evans and Yuan (2020) (associated blog post) for discussions of why smaller MDEs may be more realistic for education interventions.  
• Alternatively, the sample size (N) may be determined by the study budget or a partner’s constraints (e.g., number of eligible students in the partner’s schools, the number of health clinics in the district, etc.). 
• Number of clusters (k) and size of each cluster (m): It is typically cheaper to interview one more person in an existing cluster than add a new cluster due to logistical constraints (transportation, personnel costs etc.). However, the marginal cost decrease from one additional unit in an existing cluster should be weighed against the decreasing marginal power from each additional unit. The optimal number of clusters and the size of each cluster will also change based on the expected ICC. 
• Treatment allocation (P) can be manipulated to maximize power or to minimize costs. As written above, power is typically maximized with an equal allocation of units to treatment and control groups, but it is costlier to provide the treatment to more units. As a result, it’s possible that an unequal treatment allocation (e.g., 30% treatment 70% control) with a larger total sample size will be better powered than a 50/50 split with a smaller sample size. It is worth testing different sample sizes and treatment allocations given budget constraints to find the design that maximizes power. See chapter 6 of Running Randomized Evaluations for a guide on adjusting the allocation ratio to different practical scenarios.

Calculated using data such as from a baseline or pilot, existing studies, or publicly available data:

• Outcome variance (σ²) is ideally found in baseline data. If that is not available, it may be calculated using other data in the same or a similar population, or found in papers from studies on similar topics, preferably among similar populations. Useful data sources include the J-PAL/IPA Dataverses, administrative data, LSMS data, etc.⁵ Note that it is typically assumed that the outcome variance is the same across treatment arms.

• ICC: Similarly, the ICC can be estimated using data from baseline data, surveys, and papers--all preferably in the same or a similar population. See McKenzie (2012) for an explanation of one of the ways to calculate ICC based on baseline data in Stata.

• Take-up, compliance, and attrition: Literature from previous programs similar to the one under study can give a rough idea of plausible levels of take-up. Note that such literature need not be restricted to academic papers but can also include white and gray literature, reports, etc.

Initial calculations

Initial calculations should focus on whether the study may be feasible. After a proposed project passes a basic test of feasibility, power calculations can be refined and updated with new data. In performing initial calculations, keep the following in mind:

• Run initial calculations with the best data that is readily accessible, and avoid getting bogged down seeking access to non-public data or sorting out details that can be adjusted.
• You may not need to find data for each component to run power calculations. From sensitivity analysis (below) you will learn which inputs have the largest impact on power. Focus the team’s efforts on finding good estimates for the inputs that matter most.  
• Similarly, if you are missing information on certain power components, make reasonable assumptions to check potential feasibility of the study. For example, use the rules of thumb for effect sizes listed above: a 0.2 standard deviation increase is a small effect size; use this as a starting MDE and calculate the necessary sample size to determine basic feasibility.
• Perform sensitivity analysis to test how power changes with changes to any critical assumptions. Focus particularly on:
  • Key assumptions such as minimum effect size, take-up rates and intracluster correlation. Be realistic and, if anything, overly pessimistic about take-up rates.
  • Any information that was assumed due to missing data (e.g., effect sizes as in the bullet above).
  • Areas where there is some leeway for adjusting the design, such as adding units to clusters instead of vice versa, shifting treatment allocations, etc.
• Consider both analytical and simulation methods. Simulated calculations are especially helpful for more complex study designs (McConnell and Vera-Hernandez, 2015). They involve: 1) creating a fake dataset with a pre-determined effect size and design features, 2) testing the null hypothesis, 3) repeating steps 1 and 2 many (usually 1,000 or more) times, and 4) dividing the total number of rejections from step 2 by the total number of tests to calculate power. With good data on the study population, they can also be used to calculate simulated confidence intervals around null effects, or in power calculations for a small sample where some of the parametric assumptions about probability distributions may not hold. Sample code for both approaches can be found on our GitHub.

Refining calculations

If the study looks promising based on initial calculation, the next steps are to iterate over details of the study design and refine calculations to get a more precise estimate of the necessary sample size. Note that refinements may be minimal or not even necessary at all. For example, there may be no need to iterate further if the sample size is fixed and initial calculations made with reliable and relevant data showed that the design is powered under a wide range of assumptions about the ICC, take-up, etc. On the other hand, there are two key assumptions where refinements may be particularly helpful:

• Significant changes in design from what was assumed in initial calculations—such as changing the number of treatment arms, changing intake processes (which might affect take-up), changing the unit of randomization, or deciding you need to detect effects on particular subgroups—should inform, and be informed by, estimates of statistical power.

• Better estimates of key inputs: If the study has passed a basic feasibility test but there were first-order inputs for which you were unable to find satisfactory estimates for initial power calculations, it may be worth seeking additional data to refine power estimates. Consider requesting detailed operational data from the partner, or requesting non-public survey or administrative data from a third party.

If not included in initial calculations, there are additional design factors to consider when refining calculations, described further below.

Additional design factors to consider

For more information about the underlying theory of how design choices, interest in different kinds of effects, or the use of binary outcome variables influence power calculations, see McConnell and Vera-Hernandez (2015), Athey and Imbens (2017), Duflo et al. (2007), or Baird et al. (2018). Tips for incorporating such design decisions into power calculations are described below.

Binary outcomes: The outcome of interest may be a binary variable, such as whether a person is employed after a job-training program, whether a student passed a grade after a tutoring program etc. While the intuition for determining the MDE and sample size is the same as for a continuous variable, the effect size and the variance of the outcome variable changes based on the representation (log odds ratio or difference in probability) and calculation (logistic or linear regression) of the effect. See section 5 of McConnell and Vera-Hernandez (2015) for a statistical representation of power and sample code for a binary outcome variable.  

Unequal cluster size: Clusters will often be of unequal sizes due to real-world constraints in splitting units--this can include both an unequal number of clusters in each treatment arm as well as an unequal number of units within clusters. For example, a study randomizing at the teacher level and measuring outcomes at the student level will likely have different size clusters. Unless there is wide variation in cluster sizes, the total sample size required will be only slightly higher with unequal clusters than it would be with equal clusters. See page 12 of McConnell and Vera-Hernandez (2015) for more.

Stratified randomization: Though randomization removes ex ante systematic bias between the treatment and the control groups, researchers may choose to stratify random assignment to ensure that the treatment and the control groups have a similar proportion of units with certain important characteristic(s).  Since the stratification variable is likely to be correlated with the outcome variable, stratification will typically reduce the residual variance of the outcome variable, increasing power. See J-PAL's randomization resource and Athey and Imbens (2017) for more information on stratification. 

Heterogeneous treatment effects: Stratification can also be useful for calculating the heterogeneous effects of the treatment among certain groups. For example, we may be interested in comparing the heterogeneous results among men versus women, among college-educated vs non-college educated voters, etc. In this case, it is useful to conduct separate power calculations for the two groups to ensure that the study is sufficiently powered to detect the heterogeneous treatment effects of interest. 

Adding covariates: Similar logic as with stratification applies here: randomization precludes the necessity of including covariates for unbiased estimates of the treatment effect but doing so may be useful to reduce the residual variance of the outcome variable. All else equal, this increases power, though the extent of this depends on the size of the dataset and the correlation between the covariates and the outcome variable, among other factors.⁶ See J-PAL's data analysis resource for guidance on including covariates in your analysis and Athey and Imbens (2017) for more. 

Multiple treatment arms: If the study design includes multiple arms, we may be interested in pairwise comparisons between the treatment arms or between one treatment arm and the control. Ensure that the study is powered for the smallest MDE among the comparisons that will be made (though this can significantly increase costs if comparing two similar treatments). It can be helpful to first calculate the MDE for each pairwise comparison assuming equal allocation fractions, then recalculate the MDEs after rearranging the samples in different ways to understand which comparisons can be meaningfully made within the budget constraint. Note that if in analysis you will adjust for making multiple comparisons between groups (e.g., control and treatment 1, control and treatment 2, treatments 1 and 2, etc.) you should do the same in power calculations.

Multiple outcomes: As with multiple treatments, if the study will look at multiple outcomes, it is important to ensure that it is powered to detect all outcomes, i.e., pick the sample size and design that is powered to detect the smallest MDE of the outcomes of interest.

Multiple hypotheses: When evaluating multiple hypotheses, it is important to note that significance level or the probability of rejecting the null hypothesis increases with the number of hypotheses. This can be dealt with by adjustments for multiple hypothesis testing (MHT), such as with the Bonferroni or Benjamini Hochberg adjustments. Power may increase or decrease based on the type of adjustment, number of hypotheses and the correlation between the null hypotheses (Porter 2016; McConnell and Vera-Hernandez 2015), but if you will adjust for MHT in analysis it is important to do so in power calculations to ensure the study is adequately powered. See J-PAL’s data analysis resource as well as EGAP’s methods guide and McKenzie (2020) for a longer description of the problem, section 4 of List, Shaikh and Xu (2016) for a demonstration of multiple hypothesis adjustments with multiple treatment arms and heterogenous effects, and pages 29-30 of McConnell and Vera-Hernandez (2015) for a description of applications to  power calculations.

Interpreting the calculated MDE

For a continuous outcome variable, the minimum effect size is measured in the same units as the outcome variable, unless it is a standardized effect size (i.e., the effect size is divided by the outcome standard deviation). In this case, the MDE is expressed in terms of standard deviations. Mechanically, this is done by rearranging the variance term in equation 1.

If the outcome variable is a binary variable, the effect size may be measured as the difference in the log odds ratio or in the probability of success between the treatment and the control. For the latter, the MDE is expressed in percentage points (e.g., an MDE of .06 is interpreted as a 6 percentage point increase over the control group mean). Note that the built-in Stata command power calculates the latter (probability of success) measure.

It is important to remember that statistical power calculations are based on the underlying sampling distribution of the effect estimate. So even if the study has 80 percent power, there is still a 20 percent chance that you might not detect the effect if the program has an impact. Conversely, an underpowered study may still detect an effect if it is true but with a lower probability.  The next section discusses possible adjustments if the study is underpowered. 

Troubleshooting: Underpowered designs

If the study is underpowered but may be feasible, try to tweak the design or increase the sample size. Some ideas are listed below.

• If budget is the limiting factor: Consider not doing a baseline (McKenzie 2012), changing the treatment allocation (add units to the control group and reduce the number of units in the treatment group), adding units instead of clusters, simplifying the design (e.g., dropping a treatment arm, limiting pairwise comparisons between similar treatments, etc.). See the discussion starting on page 21 of McConnell and Vera-Hernandez (2015) for a statistical framework and sample simulation code for cost optimization. 
  • Note that stratification has an ambiguous effect on power. As mentioned above, adding covariates or stratifying increases the precision of treatment effect estimates, thereby reducing the total necessary sample size. However, the sample size required to detect heterogeneous treatment effects by stratum will likely be higher. If limited by budget, it is worth testing how stratification affects the study’s power.
• If sample size is the limiting factor: An obvious option is to simplify the design, say by reducing the number of treatment arms or focusing on a smaller set of the most important primary outcomes. As above, the inclusion of covariates can increase power, though if the sample size is fixed and is the limiting factor, you would unlikely be able to detect heterogeneous treatment effects. An alternative, if not limited by budget, is to increase the number of rounds of data collection (McKenzie 2012). 

If the study remains underpowered, even with any feasible design tweaks, then the risks of going forward with the study outweigh the benefits, and you should hold off on moving forward until the sample size can be increased, more data is available, etc. At best, an underpowered study is a waste of money—you won’t be able to determine whether the program is effective. At worst, null results from an underpowered study may be used to justify ending a program if they are conflated with the program itself being ineffective. More on this can be found in J-PAL North America’s guide.

See our GitHub page for sample Stata and R code for conducting power calculations using built-in commands and by simulation.

Last updated August 2021.

These resources are a collaborative effort. If you notice a bug or have a suggestion for additional content, please fill out this form.