Is Your Study Big Enough? A Guide to Statistical Power

Sample Size Estimation answers how many people do I need to estimate this value precisely. Power Analysis answers a related but different question: if I run this specific statistical test, how likely am I to detect a real effect, or how many people do I need to have a good chance of detecting it. This guide first explains what power means and how to choose it, then walks through how to apply it in AnalyZ Solutions.

01What statistical power and significance level mean

Power, formally 1 minus beta, is the probability that a study correctly detects an effect that is really there. A power of 80% means that if the effect genuinely exists at the size specified, there is an 80% chance the test comes back statistically significant, and a 20% chance of missing it entirely, purely due to sample size.

Significance level, alpha, is the counterpart risk. It is the probability of concluding an effect exists when it actually does not, a false positive. A significance level of 0.05, corresponding to 95% confidence, is the standard default across most fields.

How to choose these values

80% power and a 0.05 significance level are conventional defaults, not fixed rules. Consider raising power to 90% when missing a real effect would be especially costly, for example when the result determines whether a programme continues. Consider tightening significance level below 0.05 when a false positive would be especially costly, for example before recommending a policy change to a wide population based on the result. Relaxing either value below its convention is sometimes acceptable for lower stakes, exploratory work, but should be a deliberate choice rather than a default.

02Two ways to use each tool

Every test in Power Analysis works in either direction.

• Achieved power. A sample size is already known or fixed, and the tool reports the power that sample size provides. Useful when working backward from a fixed budget or an existing dataset.
• Required sample size. A target power is specified, usually 80% or 90%, and the tool reports the minimum sample needed to reach it.

After calculating, a power curve chart shows how power changes across a range of sample sizes for small, medium, and large effects, a fuller picture than any single number can give.

03Choosing the right test

04Understanding effect size

Every test requires specifying how large an effect should be detectable, expressed in standardised units so it is comparable across different outcome scales. Cohen's benchmarks are the field standard.

When there is no prior basis for the effect size, no pilot data, no comparable published study, a medium effect is the standard planning default. Incidence rate ratio does not have an equivalent standardised Cohen benchmark, since a rate ratio is already expressed in directly interpretable terms, for example an IRR of 1.5 meaning a rate 50% higher in one group. Base this figure on prior surveillance data or published literature for a comparable outcome, rather than a generic small, medium, or large preset.

05Logistic regression's extra rule, EPV

Logistic regression has a second constraint beyond power, Events Per Variable, the number of outcome events divided by the number of predictors in the model.

A conventional minimum of 10 events per predictor helps avoid an unstable, overfit model, one whose estimated relationships would likely look quite different if the study were repeated with a new sample. A model with 5 predictors, for example, needs at least 50 outcome events, not 50 total participants, to meet this minimum. Since the total sample required to accumulate a given number of events depends on how common the outcome is, a rarer outcome requires a much larger total sample to reach the same EPV.

The logistic regression tool calculates the required sample from both constraints, the power-based minimum and the EPV-based minimum, and reports whichever is larger as the binding requirement. It also reports the actual EPV implied by the result, so it is visible which constraint determined the final sample size.

1. Choose the test matching your planned analysis. Select from the nine tests listed above, grouped by what you are comparing: means between groups, a categorical association, a change measured twice on the same subjects, a regression model, or two incidence rates. If you are unsure which test fits your design, the "Choosing the right test" table above maps each one to a specific use case.
2. Choose achieved power or required sample size. Select achieved power if your sample size is already fixed, by budget, an existing dataset, or another constraint, and you want to know what power that gives you. Select required sample size if you are still planning and want to work forward from a target power to the minimum n needed.
3. Set your significance level and effect size. 0.05 is the standard significance level default. For effect size, use a value calculated from pilot data or comparable published research if one is available, or one of Cohen's small, medium, or large benchmarks otherwise, as discussed above. This single input drives the result more than any other, so treat it as a genuine assumption worth documenting, not an arbitrary setting.
4. For logistic regression, also enter the number of predictors and your target EPV. The tool calculates both the power-based minimum sample size and the EPV-based minimum, and reports whichever is larger as the binding requirement, along with the actual EPV implied by that result.
5. For incidence rate ratio, enter your rates in events per person-year, and person-time rather than a headcount. Enter both rates directly, or one rate plus the incidence rate ratio to detect. In achieved power mode, enter the person-time already available per group; in required sample size mode, the tool solves for the person-time needed instead. This test assumes equal person-time contributed to each group.
6. Calculate. In achieved power mode, the result shows the power your specified sample size provides. In required sample size mode, it shows the minimum n needed to reach your target power, at your chosen significance level and effect size.
7. Review the power curve. After calculating, view the chart showing how power changes across a range of sample sizes for small, medium, and large effects, with your own inputs marked. This makes visible how sensitive your result is to the effect size assumption, not just to the sample size itself.

06Worked example

An evaluation plans to compare average scores on a knowledge test between two independent groups, using a two-sample t-test. Based on similar studies, a medium effect, Cohen's d of 0.5, is a realistic planning assumption. The team wants 80% power at a significance level of 0.05.

Running the same inputs in achieved power mode, entering 64 per group directly, confirms the result reaches approximately 80% power at this effect size. The power curve chart shows that the same sample would achieve noticeably lower power against a small effect, Cohen's d of 0.2, and higher power against a large effect, Cohen's d of 0.8, illustrating why the effect size assumption drives the result as much as the sample size itself.

07Common mistakes to avoid

• Choosing an optimistic effect size rather than a realistic one. Powering a study around a large effect when the true effect is more likely small or medium produces a study with far less power than intended, in practice.
• Using an independent groups test when the design is paired. If the same individuals are measured twice, use the paired t-test or McNemar's test, not the two-sample or chi-square versions, which understate the available power for a paired design.
• Ignoring EPV when planning a logistic regression model. A sample sized only from the power calculation can still leave the model unstable if the number of events relative to predictors is too low. Check both constraints, not just power.
• Calculating power after the study is already complete to explain a null result. This, sometimes called post-hoc power, is not informative and is generally discouraged. Power should be calculated and justified before data collection, as part of planning the study.
• Using incidence rate ratio when follow-up time is actually fixed and equal for everyone. If every participant is observed for the same period, a proportion-based test such as chi-square or the two-sample t-test is simpler and equally valid. Reserve incidence rate ratio for genuinely variable follow-up time.