2024-06-23

What p-value can tell

In statistical hypothesis tests, the null hypothesis basically states that it is not extraordinary

For example, if a test is comparing the difference between two groups, the null is that there is no difference. When a test aims to see if a factor has a real impact on something, then the null is that there is no effect. 

The null means what happened in the outcome is not extraordinary, and it just happened due to a random fluctuation. The apparent difference in the two groups could be just a fluke. The seemingly impactful outcome could just happen by chance.

Whether the outcome happened by chance or not is decided by a p-value.

A p-value provides a quantity to tell how much the outcome is probable when the null is right. As far as the p-value is less than 5% (a conventional threshold), the null hypothesis can be rejected

Now that the null cannot explain the outcome, we can say that it is extraordinary: the difference is not a fluke, and the impact is not by chance, statistically speaking, it is statistically significant