Test Statistic Calculator

A test statistic measures how far a sample result sits from what the null hypothesis predicts, in units of the result's own uncertainty. For a mean that is one line: the difference between the sample mean and the hypothesized mean, divided by the standard error. Everything else this calculator reports follows from that number and the distribution it is read against.

Take the default. A filling machine is meant to put 500 g in each pack; a sample of 25 packs averages 497.2 g with a standard deviation of 6.1 g. The standard error is 6.1 divided by the square root of 25, which is 1.22 g: how much a mean of 25 packs moves from sample to sample. The shortfall of 2.8 g is 2.8 / 1.22 = 2.295 standard errors, so t = -2.295. Read against Student's t distribution on 24 degrees of freedom, a result that far from 500 in either direction turns up about 3.1% of the time if the machine is in fact right. That is below 0.05, so at that level the null is rejected.

z or t. A z test reads the statistic against the normal distribution and is right only when the population standard deviation is known in advance. Outside textbooks and a few calibrated processes, it is not. When the SD is estimated from the same sample, the estimate itself wobbles, and Student's t distribution accounts for that with fatter tails. The difference is real at small n: the two-tailed 5% critical value is 2.064 for t on 24 df against 1.960 for z. By about a thousand observations the two agree to two decimals, so using t whenever the SD came from the data is never wrong.

Degrees of freedom are the sample size less the number of quantities estimated before the spread could be measured. A one-sample or paired test spends one on the mean, so df = n - 1.

Two samples. Comparing two group means, the standard error combines both groups' variances. Welch's test lets each group keep its own variance and computes a fractional df from the Welch-Satterthwaite formula. The older Student version pools the two variances into one, assuming they are equal. When the groups have different spreads and different sizes, the pooled test can be badly wrong in either direction, while Welch's loses almost nothing when the variances do happen to match. That is why it is the default in R's t.test. This calculator shows both. On the default two-sample inputs they disagree: pooled gives p = 0.042 and clears 0.05, Welch gives p = 0.055 and does not, because the smaller group has the larger spread and pooling understates the uncertainty that brings.

Paired data (the same people before and after, or two measurements on each unit) is not two samples. Subtract within each pair and run a one-sample t test on the differences. Pairing removes the person-to-person variation, which is usually most of it, so a paired test on the same data is far more sensitive than a two-sample one. It needs the standard deviation of the differences, not of either set of measurements.

One tail or two. A two-tailed test counts a result in either direction as evidence. A one-tailed test puts all of alpha in one tail, so it rejects with a smaller statistic and is blind to an effect in the other direction. It is legitimate only when that direction was decided before the data were seen and an effect the other way would lead to the same action as no effect.

What the p-value is not. It is the probability of data at least this extreme if the null were true. It is not the probability that the null is true, not the probability the result is a fluke, and not a measure of how large or important the effect is. A big sample makes trivial differences significant; a small one leaves real differences undetected. That is why the page also gives the confidence interval (the range of values the data are consistent with) and an effect size, which measures the difference in standard deviations of the data and does not grow with n. A result can be statistically significant and practically meaningless, and a non-significant result is not evidence that there is no effect.

The other modes cover the neighboring questions. A z score places a single value against a mean and SD, with its percentile under a normal curve. The p-value mode reads any z, t or chi-square statistic you already have; chi-square tests of fit and independence are upper-tailed by construction. For a difference between two conversion rates or other proportions, use the A/B test calculator, which runs the two-proportion z test.

The distributions are computed directly rather than looked up: the normal by Hart's rational approximation and a continued fraction in the tail, t through the regularized incomplete beta function, and chi-square through the incomplete gamma function. Each tail is computed as itself rather than as one minus the other, so a p-value of 1e-12 is reported as that and not as zero.