BABS3281 P-value Interpreter

This tool accompanies the statistics lectures for BABS3281 Molecular Frontiers. It calculates the expected proportion of statistical tests with true/false significant results given: (1) the p-value cutoff, (2) the statistical power and (3) the real-world likelihood that an experiment is “positive”, i.e. the null hypothesis is not true.

Changing the values below will update the table to the right. See below the table for descriptions of the parameters and the output.

© 2017 Richard J. Edwards
An EdwardsLab Shiny App
Version 0.2.0 | GNU GPL v3

Parameters:

  • P-value cutoff: The stringency of your statistical test. This sets the False Positive Rate.
  • Statistical power: The statistical power of your test. This is the probability of a significant result at the given P-value cutoff if there really is a difference (e.g. between treatment and control) of the desired size. This sets the False Negative Rate.
  • Real % chance of difference: The real world determines the a priori likelihood that your experiment should have a positive (“significant”) result, i.e. the null hypothesis is not true.

Table above calculates the different possible outcomes based on these three parameters. The Correct and Incorrect columns give you the probability of being correct if you trust the result of a statistical test in this scenario.

Background

Interpreting p-values is more complex than at first it appears. A p-value tells you the False Positive Rate of your statistical test if the null hypothesis is true. In other words, you only expect the actual False Positive Rate to match the p-value if there is no chance of there being a real effect. If you expect a real effect, your interpretation of p-values changes. This tool is designed to help with this intepretation. In reality, you probably do not know this number, in which case this tool is a cautionary tale regarding the importance of this unknown aspect of your experiment.

The questions researchers really want to answer are when presented with a p-value are:

  • What proportion of “significant” results represent a real effect? (True Positives)
  • What proportion of “non-significant” results should have actually been significant? (False Negatives)

These values can be found in the Correct and Incorrect columns.

Examples

See the GraphPad Statistics Guide for some example scenarios.