Power Analysis evaluates the probability a statistical test will detect a real effect. This guide explains the underlying statistics first, then shows how to apply them using AnalyZ Solutions.
Sample Size Estimation answers how many people do I need to estimate this value precisely. Power Analysis answers a related but different question: if I run this specific statistical test, how likely am I to detect a real effect, or how many people do I need to have a good chance of detecting it. This guide first explains what power means and how to choose it, then walks through how to apply it in AnalyZ Solutions.
Power, formally 1 minus beta, is the probability that a study correctly detects an effect that is really there. A power of 80% means that if the effect genuinely exists at the size specified, there is an 80% chance the test comes back statistically significant, and a 20% chance of missing it entirely, purely due to sample size.
Significance level, alpha, is the counterpart risk. It is the probability of concluding an effect exists when it actually does not, a false positive. A significance level of 0.05, corresponding to 95% confidence, is the standard default across most fields.
80% power and a 0.05 significance level are conventional defaults, not fixed rules. Consider raising power to 90% when missing a real effect would be especially costly, for example when the result determines whether a programme continues. Consider tightening significance level below 0.05 when a false positive would be especially costly, for example before recommending a policy change to a wide population based on the result. Relaxing either value below its convention is sometimes acceptable for lower stakes, exploratory work, but should be a deliberate choice rather than a default.
Every test in Power Analysis works in either direction.
After calculating, a power curve chart shows how power changes across a range of sample sizes for small, medium, and large effects, a fuller picture than any single number can give.
| Test | Use when | Effect size |
|---|---|---|
| Two-sample t-test | Comparing means between two independent groups | Cohen's d |
| One-sample t-test | Testing whether a group mean differs from a known reference value | Cohen's d |
| Paired t-test | Comparing two measurements on the same individuals, pre and post | Cohen's dz |
| Chi-square test | Testing association between two categorical variables | Cohen's w |
| McNemar's test | Detecting a change in a binary outcome measured twice on the same subjects | Discordant pair proportions |
| One-way ANOVA | Comparing means across three or more independent groups | Cohen's f |
| Linear regression | Detecting a given R² with one or more predictors | Cohen's f² |
| Logistic regression | Binary outcome model with multiple predictors | Odds ratio and EPV |
Every test requires specifying how large an effect should be detectable, expressed in standardised units so it is comparable across different outcome scales. Cohen's benchmarks are the field standard.
| Effect size | Small | Medium | Large |
|---|---|---|---|
| Cohen's d, t-tests | 0.2 | 0.5 | 0.8 |
| Cohen's w, chi-square | 0.1 | 0.3 | 0.5 |
| Cohen's f, ANOVA | 0.10 | 0.25 | 0.40 |
| Cohen's f², regression | 0.02 | 0.15 | 0.35 |
When there is no prior basis for the effect size, no pilot data, no comparable published study, a medium effect is the standard planning default.
Logistic regression has a second constraint beyond power, Events Per Variable, the number of outcome events divided by the number of predictors in the model. A conventional minimum of 10 events per predictor helps avoid an unstable, overfit model. The logistic regression tool reports both the power-based and EPV-based minimum sample size, and uses whichever is larger.
A short screen recording showing these steps in the AnalyZ Solutions interface can be embedded here.
Open Power Analysis in AnalyZ Solutions. Free, browser based.
Try it out