← Evaluate statistically based reports

6 Statistical significance vs practical importance

These sound similar but are genuinely different questions — a real report can be statistically solid and still practically meaningless, or vice versa in smaller samples.

How to tell them apart

  • Statistical significance asks: is this result unlikely to be due to chance alone? A confidence interval that clears a value (or excludes zero difference) is evidence of significance.
  • Practical importance asks: does the SIZE of the effect actually matter for a real decision? A tiny, reliably-detected difference can still be irrelevant.
  • With a large enough sample, almost ANY tiny true difference becomes statistically detectable — so a huge sample size should make you MORE cautious about practical importance, not less.
  • Always ask what a genuinely large or meaningful effect would look like in this context, and compare the actual effect size to that benchmark.
Exam-style A statistically significant but practically trivial result A drug trial with 50,000 participants finds it lowers blood pressure by 0.4 mmHg, with a confidence interval that excludes zero. Evaluate the claim that the drug is effective. The huge sample size means even a genuinely tiny effect (0.4 mmHg) can be detected reliably — the confidence interval excluding zero shows statistical significance. But 0.4 mmHg is far below any clinically meaningful change in blood pressure, so the result has little PRACTICAL value for patients. The claim "the drug is effective" should be qualified: statistically real, but not practically significant at this dose. ✓ A confident, real, and utterly trivial effect — exactly the gap between statistical significance and practical importance a strong evaluation names explicitly. ✕

Treating "statistically significant" as meaning "important"

A report headline calling a result "significant" is using the STATISTICAL sense (unlikely due to chance), which says nothing about whether the effect size actually matters. Always check the actual magnitude of the effect before accepting that a "significant" finding is meaningful in practice.

Practise just this