Online Tool Store
· 5 min read
How to Run a One-Way ANOVA Properly
Heshan Fernando
Co-founder & COO
Three treatment groups, a p value of 0.0007, and a conclusion in the report saying group B outperformed the others.
The p value does not say that. It says at least one group differs from at least one other, which is a considerably weaker statement than the one that got written down.
What the F statistic compares
ANOVA asks whether the spread between group means is larger than the spread within groups would lead you to expect.
F = variance between groups ÷ variance within groups
If the groups are really the same, their means still differ a little through sampling noise, and that noise is estimated by the within-group variance. A large F means the means are further apart than that noise explains.
The logic is the same as a t-test generalised to more than two groups, which is why for exactly two groups ANOVA and a t-test give identical conclusions.
Why not run several t-tests
The obvious alternative is comparing every pair separately. With three groups that is three tests; with five groups it is ten.
Each test carries its own false positive risk. Run enough of them and finding at least one spurious “significant” result becomes likely rather than unlikely — with ten tests at the conventional threshold, the chance of at least one false positive is substantial.
ANOVA tests all groups in one procedure, which controls that inflation. It is the reason the method exists.
The post-hoc test is not optional
A significant F tells you the groups are not all the same. It does not identify which differ, and with four groups there are six possible pairs.
Post-hoc tests — Tukey’s HSD being the common one — compare the pairs while correcting for the multiple comparisons being made. That correction is what keeps the overall false positive rate controlled, and it is why you cannot simply run t-tests after the ANOVA and report those.
Reporting “the ANOVA was significant, therefore B is best” without a post-hoc test is the most common error in this analysis, and it is usually made by someone who did the harder part correctly.
| Output | Tells you |
|---|---|
| F statistic and p | At least one group differs |
| Effect size (eta squared) | How much of the variance is between groups |
| Post-hoc pairwise | Which groups differ |
Report the effect size
A p value answers whether a difference is distinguishable from noise. It says nothing about how large it is, and with a big enough sample a trivial difference is significant.
Eta squared gives the proportion of total variance attributable to group membership. An eta squared of 0.80 means group membership explains most of the variation; 0.02 means it explains almost none, however small the p value.
Both belong in the report. A significant result with a tiny effect size is a real finding that does not matter.
Plan the comparisons before collecting data
A distinction that determines what the analysis can honestly claim.
Planned comparisons are decided before seeing the data, based on the hypothesis. They are more powerful statistically and they do not require the same correction as searching.
Post-hoc comparisons are decided after seeing which groups look different. They require correction precisely because the comparison was chosen by looking at the data, which is a form of multiple testing even when only one test is run.
Deciding afterwards and reporting it as though it were planned is a well-known way to produce results that do not replicate.
Writing the intended comparisons down before collecting anything takes minutes and it is what separates a test of a hypothesis from a search for a significant result.
Common mistakes to avoid
- Concluding which group is best from the ANOVA alone.
- Running unadjusted t-tests as the follow-up, which reintroduces the problem ANOVA solved.
- Reporting p without effect size.
- Ignoring the assumptions — independent observations, roughly normal residuals, similar variance across groups. Very unequal variances need Welch’s ANOVA.
- Using ANOVA on repeated measurements from the same subjects, which violates independence and needs a repeated-measures design.
How to do it with ANOVA Calculator
The ANOVA Calculator reports the F statistic, p value and effect size together.
- Enter each group’s values on its own line; groups need not be equal in size.
- Read F and p, then the effect size.
- If significant, run a post-hoc test to find which pairs differ.
- Check the assumption notes, particularly if group variances look very different.
Other statistics tools are in the tools directory.
Frequently asked questions
What does a significant ANOVA tell me?
That at least one group mean differs from at least one other. It does not say which, or how many, or by how much. A post-hoc test identifies the pairs with a correction for multiple comparisons.
Why not just run several t-tests?
Because each test carries a false positive risk and running many inflates the overall chance of a spurious result. ANOVA tests all groups at once, controlling that.
What assumptions does it make?
Independent observations, roughly normal residuals and similar variance across groups. It tolerates mild violations; very unequal variances or group sizes call for Welch’s ANOVA or a non-parametric alternative.
Final thought
Report the effect size next to the p value, and never name a winning group without a post-hoc test. Both are one extra step and they are the difference between a defensible result and an overstated one.