Unmasking P-values

Take Home Messages:

• P-values say nothing about effect size, look up the confidence interval and point estimate to address clinical relevance of any “findings”
• The P-value does not measure the likeliness of H0 being true
• Rejection of the null hypothesis does not mean that the true effect is as specified in the alternative hypothesis that the study was powered for
• H0 cannot be proven. A non-significant P-value does not prove that the null hypothesis is true
• P-values are influenced by sample size (power).
• There is always a chance that a statistically significant finding be a false positive
• The chance of false positive findings increases sharply with the number of tests

References

Amrhein V, Greenland S, McShane B.  Scientists rise up against statistical significance. Nature 567, 305-307 (2019).doi: 10.1038/d41586-019-00857-9

Cristea IA, Ioannidis JPA. P values in display items are ubiquitous and almost invariably significant: A survey of top science journals. PLoS One. 2018;13(5):e0197440. Published 2018 May 15. doi:10.1371/journal.pone.0197440

Fisher RA. Statistical Methods for Research Workers. Edinburgh, UK: Oliver and Boyd; 1925.

Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence  intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31:337–350.

Goodman S. A dirty dozen: twelve p-value misconceptions Semin Hematol. 2008 Jul;45(3):135-40. doi: 10.1053/j.seminhematol.2008.04.003

Ioannidis JPA. The Proposal to Lower P Value Thresholds to .005. JAMA. 2018;319(14):1429‐1430. doi:10.1001/jama.2018.1536

Ioannidis JPA. Publishing research with P-values: Prescribe more stringent statistical significance or proscribe statistical significance?. Eur Heart J. 2019;40(31):2553‐2554. doi:10.1093/eurheartj/ehz555

Neyman J, Pearson E. On the problem of the most efficient tests of statistical hypotheses. Philos Trans R Soc Lond A. 933;231:289–337.

Schroeber P, Bossers SM, Schwarte LA. Statistical Significance Versus Clinical Importance of Observed Effect Sizes: What Do P Values and Confidence Intervals Really Represent?. Anesth Analg. 2018;126(3):1068‐1072. doi:10.1213/ANE.0000000000002798

Szucs D, Ioannidis JPA. When null hypothesis significance testing is unsuitable for research: a reassessment. Front Hum Neurosci.2017;11:390.

Wasserstein RL, Schirm AL and Lazar NA (2019) Moving to a World Beyond “p < 0.05”, The American Statistician, 73:sup1, 1-19, DOI: 10.1080/00031305.2019.1583913