Key takeaways
- A hypothesis test asks how surprising your data would be if the null hypothesis were true; the p-value quantifies that surprise.
- A p-value is not the probability that the null hypothesis is true, and p > .05 does not prove there is no effect.
- A 95% confidence interval is estimate ± critical value × standard error, and gives a range of plausible values for the population parameter.
- Type I errors (false positives) are controlled by alpha; Type II errors (missed effects) are reduced by larger samples and higher power.
- In APA style, report confidence intervals in square brackets: M = 102.50, 95% CI [98.44, 106.56].
What is hypothesis testing?
Hypothesis testing is a procedure for deciding whether sample data provide enough evidence against a default assumption, the null hypothesis, to reject it. You calculate a test statistic, find the probability of a result at least that extreme if the null were true (the p value), and compare it with a threshold you set in advance (alpha, usually .05).
A confidence interval answers the complementary question: given the data, what range of values for the population parameter is plausible? Modern reporting standards, including APA 7, expect both.
The five steps of a hypothesis test
- State the hypotheses. The null hypothesis (H₀) usually says there is no effect or no difference: μ₁ = μ₂, ρ = 0, μ = 100. The alternative (H₁ or Hₐ) says there is: μ₁ ≠ μ₂ (two-tailed) or μ₁ > μ₂ (one-tailed, only if specified in advance).
- Choose alpha (α). The risk of a false positive you will accept. α = .05 is conventional; some fields use .01 or adjust for multiple tests.
- Pick the test and check its assumptions. The test depends on your question, variables and design; see our guide to choosing a statistical test.
- Compute the test statistic and p value. Most statistics take the form (estimate − null value) / standard error.
- Decide and interpret. If p ≤ α, reject H₀; otherwise, fail to reject it. Then describe the size and direction of the effect, not just the decision.
What a p-value does and does not mean
The p value is the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true and the test’s assumptions hold.
| A p-value is not… | Why |
|---|---|
| The probability that H₀ is true | It is calculated by assuming H₀ is true, so it cannot also be the probability of H₀. |
| The probability your result is due to chance | Same problem, phrased differently. |
| A measure of effect size | A tiny, unimportant effect can give p < .001 in a very large sample. |
| Proof of no effect when p > .05 | A small or noisy study may simply lack the power to detect a real effect. |
That is why a p value should always travel with an effect size and a confidence interval. The American Statistical Association’s 2016 statement on p values makes the same point: no single threshold should decide whether a finding matters.
Type I and Type II errors, and power
| H₀ actually true | H₀ actually false | |
|---|---|---|
| Reject H₀ | Type I error (false positive), probability α | Correct decision, probability 1 − β (power) |
| Fail to reject H₀ | Correct decision, probability 1 − α | Type II error (false negative), probability β |
Power is the probability of detecting an effect of a given size if it exists. A target of 80% is common. Power rises with larger samples, larger true effects, less noisy measurement and a higher alpha. Running a power analysis before data collection, and reporting it in your methods, is the best defence against an inconclusive study. Running many tests on the same data inflates the overall Type I error rate; corrections such as Bonferroni or Holm keep it in check.
Want a second pair of eyes? Want your hypotheses, p-values and confidence intervals checked before they go to your supervisor? We explain each one so you can defend it. See statistics help.
Confidence intervals: formula and meaning
Most confidence intervals follow one pattern:
CI = estimate ± (critical value × standard error)
For a mean: x̄ ± t* × (s / √n), where s is the sample standard deviation and t* comes from the t distribution with n − 1 degrees of freedom. For a proportion with a large sample: p̂ ± z* × √[p̂(1 − p̂) / n].
The half-width, critical value × standard error, is called the margin of error (sometimes written E). The critical value, often called z-star (z*), depends on the confidence level:
| Confidence level | z* |
|---|---|
| 90% | 1.645 |
| 95% | 1.960 |
| 99% | 2.576 |
With small samples and an estimated standard deviation, use t* instead, which is slightly larger (2.030 for 35 degrees of freedom at 95%).
Interpretation: if you repeated the study many times and built a 95% interval each time, about 95% of those intervals would contain the true population value. Any single interval either contains it or not, so avoid saying “there is a 95% probability the true mean is in this interval”. In practice, it is fair to say the interval shows the range of values reasonably compatible with your data. If a 95% CI for a difference excludes zero, the corresponding two-tailed test gives p < .05.
A worked example: test and interval together
A school district’s standardised reading test is scaled so the national mean is 100. A researcher samples 36 pupils from a new literacy programme and finds M = 102.5, SD = 12.0. Is their mean different from 100?
- H₀: μ = 100; H₁: μ ≠ 100; α = .05, two-tailed.
- Standard error = 12.0 / √36 = 2.0.
- t(35) = (102.5 − 100) / 2.0 = 1.25.
- p = .220. Since .220 > .05, we fail to reject H₀.
- 95% CI = 102.5 ± 2.030 × 2.0 = [98.44, 106.56].
The interval shows why “not significant” does not mean “no effect”: the data are compatible with programme pupils scoring slightly below the national mean or up to about 6.5 points above it. A larger sample would narrow that range. This is a one-sample t-test; the same logic applies to every other test.
Proportion example: in a survey of 200 students, 116 (58%) say they use AI tools for brainstorming. The standard error is √(.58 × .42 / 200) ≈ .035, so the margin of error is 1.96 × .035 ≈ .068 and the 95% CI is [.51, .65], or 51% to 65%.
One-tailed vs two-tailed tests, and effect sizes
A two-tailed test looks for a difference in either direction; a one-tailed test looks in one direction only and puts all of alpha in that tail. For the same data, a one-tailed p is half the two-tailed value when the effect is in the predicted direction. That makes one-tailed tests tempting, but they are only defensible when the direction was specified before data collection and a result in the opposite direction would genuinely be treated the same as no effect. Most examiners and journals expect two-tailed tests, and switching after seeing the data is a form of p-hacking.
An effect size puts the result on a scale that does not depend on sample size. The most common are:
| Effect size | Used with | Rough benchmarks (small / medium / large) |
|---|---|---|
| Cohen’s d | Differences between two means | 0.2 / 0.5 / 0.8 |
| Pearson’s r | Correlations | .10 / .30 / .50 |
| η² or partial η² | ANOVA | .01 / .06 / .14 |
| Odds ratio | Logistic regression, 2 × 2 tables | Depends heavily on context |
The benchmarks come from Cohen and were meant as a last resort when nothing better is known. Comparing your effect with published studies in your own field is more persuasive. In the reading example, d = (102.5 − 100) / 12 ≈ 0.21: a small effect that this sample of 36 was never well powered to detect.
How to report p-values and confidence intervals in APA style
APA 7 rules worth memorising:
- Report exact p values to two or three decimals (p = .220, p = .002). Use p < .001 for smaller values; never write p = .000.
- No leading zero for p, since it cannot exceed 1.
- Put confidence limits in square brackets, separated by a comma, and state the level the first time: 95% CI [98.44, 106.56]. After that, “CI” alone is fine if the level does not change.
- When the CI appears inside parentheses, keep the square brackets inside the parentheses: (M = 102.50, 95% CI [98.44, 106.56]).
A full confidence interval statement in text:
Pupils in the literacy programme scored a mean of 102.50 (SD = 12.00), 95% CI [98.44, 106.56], which did not differ significantly from the national mean of 100, t(35) = 1.25, p = .220.
Fifty-eight percent of students (116 of 200) reported using AI tools for brainstorming, 95% CI [51%, 65%].
If you are preparing results for a thesis and want each test checked and explained, our statistics tutor sessions or statistics help can walk through them with you.