November 21

P-value

The p-value is a measure of the strength of evidence against the null hypothesis. Formally, it is the probability of observing a test statistic as extreme as, or more extreme than, the one observed, assuming the null hypothesis is true. That is,

\[p = \begin{cases} P(T > t) & \text{if } H_a: \theta > \theta_0 \text{ (Right-tailed)}, \\ P(T < t) & \text{if } H_a: \theta < \theta_0 \text{ (Left-tailed)}, \\ 2P(T > |t|) & \text{if } H_a: \theta \neq \theta_0 \text{ (Two-tailed)}. \end{cases}\]

for test statistic $t$.

Interpretation

The p-value quantifies the compatibility of the data with the null hypothesis.

Small p-value ($p \le \alpha$): The observed data is unlikely to have occurred by random chance if $H_0$ were true. This provides evidence against $H_0$.
Large p-value ($p > \alpha$): The observed data is consistent with chance variation under $H_0$. There is insufficient evidence to reject the null.

Important Note: The p-value is not the probability that the null hypothesis is true, nor is it the probability that the alternative hypothesis is false. It is strictly a probability regarding the data, given the assumption that $H_0$ is true.

The Decision Rule

To make a formal decision, compare the calculated p-value to the chosen significance level $\alpha$.

\[\text{Decision} = \begin{cases} \text{Reject } H_0 & \text{if } \text{p-value} \le \alpha \\ \text{Fail to reject } H_0 & \text{if } \text{p-value} > \alpha \end{cases}\]

If $p$ is very small (e.g., $< 0.001$), the result is often described as “highly statistically significant.”
If $p$ is between 0.05 and 0.10, the result is sometimes described as “marginally significant” (depending on the field of study).

Calculation and Distribution

The calculation of the p-value depends on the distribution of the test statistic (e.g., Standard Normal $Z$ or Student’s $t$) and the degrees of freedom.

Z-Tests: Look up the area in the tail(s) of the standard normal distribution corresponding to the calculated $z$-score.
T-Tests: Look up the area in the tail(s) of the $t$-distribution with $n-1$ degrees of freedom.

Examples

Example 1: One-Sided Z-Test (Right Tail)

A school claims that their students average higher than the national average of 500 on a standardized test. A sample of 50 students yields a mean of 515 with a population standard deviation of 40.

Step 1: State the hypotheses

$H_0: \mu = 500$
$H_a: \mu > 500$ (Right-tailed)

Step 2: Compute the test statistic

$z = \frac{515 - 500}{40 / \sqrt{50}} \approx 2.65$

Step 3: Find the p-value

We look for the area to the right of $z = 2.65$.
Using a standard normal table: $P(Z < 2.65) \approx 0.9960$.
p-value $= 1 - 0.9960 = 0.0040$.

Step 4: Conclusion

Using $\alpha = 0.05$: Since $0.0040 < 0.05$, we reject $H_0$. There is significant evidence that the students average higher than 500.

Example 2: Two-Sided T-Test

A manufacturer fills bags with 250g of coffee. A quality control manager wants to test if the bags are off-target (either underfilled or overfilled). A sample of 15 bags has a mean of 248g and a standard deviation of 4g.

Step 1: State the hypotheses

$H_0: \mu = 250$
$H_a: \mu \neq 250$ (Two-tailed)

Step 2: Compute the test statistic

$t = \frac{248 - 250}{4 / \sqrt{15}} \approx -1.94$

Step 3: Find the p-value

Degrees of freedom ($df$) = $15 - 1 = 14$.
We find the probability of being more extreme than -1.94 in either direction.
$P(T < -1.94)$ for $df=14$ is approximately $0.036$.
Because it is a two-sided test, we double this area: $\text{p-value} = 2 \times 0.036 = 0.072$.

Step 4: Conclusion

Using $\alpha = 0.05$: Since $0.072 > 0.05$, we fail to reject $H_0$. There is not sufficient evidence to conclude the bags are off-target at the 5% level.

Example 3: Comparison with Critical Value Approach

Consider the previous Example 2 (Coffee bags) where $t \approx -1.94$ and $\text{p-value} = 0.072$.

Critical Value Method ($\alpha=0.05$): The critical values for a two-tailed t-test ($df=14$) are $\pm 2.145$. Since $-1.94$ falls between $-2.145$ and $2.145$, we fail to reject.
P-value Method ($\alpha=0.05$): Since $0.072 > 0.05$, we fail to reject.

Note: Both methods will always yield the same decision regarding the null hypothesis. However, the p-value provides more granular information about the strength of the evidence than a simple “Reject/Fail to Reject” binary.