“What Does The P-Value Mean” Revisited
In a previous article, we provided a didactic definition of the p-value. However, it may not accurately reflect its full meaning.
This simplification benefits those initially engaging with the concept, helping build a foundational understanding.
We now present a more precise definition of the p-value, which requires closer attention.
Statistical Hypotheses
When conducting an inferential hypothesis test — such as chi-square, t-test, ANOVA, correlation, regression, etc. — we have two hypotheses:
NULL HYPOTHESIS (H0): This is the simpler default hypothesis. It assumes no “difference between the groups” or no “relationship between the variables.”
ALTERNATIVE HYPOTHESIS (H1): This hypothesis is the complement of H0. It suggests “differences between groups” or a “relationship between the variables.”
Significance Level And P-Value
The primary goal of any hypothesis test is to determine whether or not to reject the null hypothesis (H0). This decision depends on two key factors:
SIGNIFICANCE LEVEL (α): This is a predetermined threshold value we use to decide whether or not to reject H0. It’s typically set to 1% or 5% before performing the test.
P-VALUE (p): The p-value is a probability obtained from each inferential hypothesis test we perform.
What Does It Mean To Be Statistically Significant?
After performing our analysis and obtaining the p-value, we compare it to our pre-set significance level (α).
For example, we set a significance level (α) of 0.05 (or 5%). When comparing this α with our obtained p-value, we have two possibilities:
1. If the p-value is less than or equal to the significance level α (p ≤ 0.05), we reject the null hypothesis (H0). In this scenario, our test is considered statistically significant.
2. If the p-value is greater than the significance level α (p > 0.05), we do not reject the null hypothesis (H0). Here, our test is considered not statistically significant.
The Meaning of P-Value
In technical terms, the p-value is:
the probability of obtaining a result equal to (or more extreme than) what we observed in our data, assuming the null hypothesis is true.
For instance, if we get a p-value of 2% (p = 0.02), this means that if H0 is true, the probability of getting results equal to (or more extreme than) ours is just 2%. Since this is less than α = 5%, we reject H0.
Example: The Lady Tasting Tea
This story takes us to a summer afternoon in Harpenden, England, in the early 1920s.
A group of scientists — a statistician, an algologist, and a biochemist — had gathered for afternoon tea at the Rothamsted Experimental Station.
Muriel Bristol, the algologist, insisted that tea poured over milk tasted different from milk poured over tea.
The group challenged her assertion, finding it hard to believe there could be any noticeable difference in taste.
Ronald Fisher, the statistician, proposed an experiment. Muriel would be served eight cups of tea. Half of these would have milk poured over tea, and the other half would have tea poured over milk. However, she would not witness the preparation process.
As she tasted each cup, Muriel correctly identified the preparation method and got all eight right!
Concluding Remarks
Regarding our discussion on p-value.
Let’s consider the null hypothesis (H0) that Muriel can’t identify the different preparations. If H0 is true, the probability that Muriel could correctly identify the tea preparation in all eight cups is only 1.43%.
Since this p-value of 1.43% is less than our chosen significance level (α) of 5%, we consider the test statistically significant. Therefore, we would reject the null hypothesis (H0) and instead support the alternative hypothesis (H1) that Muriel can distinguish between the different tea preparations.
*This example illustrates the importance of setting the significance level (α) before conducting the test. Typically, α is set at either 1% or 5%. If we were to obtain a p-value between these two thresholds, we might be biased towards choosing the α that makes our result seem significant — in this case, 5%.
Remember to read the first article on p-value! Click here!
