The procedure of statistical tests
Before a statistical test is conducted, the alternative (H1) and null-hypothesis (H0) should be clearly defined as argued in the hypothesis testing section. The procedure of a statistical test and the resulting rejection of either H0 or H1 follows a general procedure where a test statistic is calculated which is then compared to a critical threshold value. If the test statistic is larger or smaller than the critical value of the statistic, depending on the test, H0 or H1 is rejected. The rejection of H0 provides support for H1 and vice verca.
Each test relies on a set of assumptions. In the t test for example, the samples involved needs to be normally distributed, have equal variances and observations must be independent. The results of the test is not reliable if the assumptions are violated. If they are, another test may be more appropriate. Parametric tests (e.g. the Z‑test, t‑test and ANOVA) are more bounded by specific assumptions than non-parametric tests (e.g. Mann-Whitney, Spearman and Kruskal Wallis test). The latter only relies on the assumption of independence. However, these tests have lower power compared to parametric tests. So, what you need to do before you proceed with a test is to check the assumptions.
The assumption of normality is easily done by a histogram, which show if the sample at hand follows a normal distribution. It does not has to be perfect. If the sample size is small, this assumption may be difficult to test. Don’t worry. Simulations have shown that parametric test in fact are quite robust to violations of this assumption. A worse scenario is though that variances are unequal. This assumption is checked by a F‑test.
The test statistic
Every test statistic is part of a mathematically defined distribution. The calculated value of a test statistic expresses its position within a distribution of possible values of the statistic if the null-hypothesis is true. If the test statistic is not likely to get within this distribution, the null-hypothesis is rejected. Which values that are “unlikely” is set by the significance level of the test (α=0.05 or 0.01). If the probability of getting the calculated value of the test statistic if the null-hypothesis holds true is less than for example 0.05, the null-hypothesis is rejected. See the Z‑test section for a detailed description of how the Z test statistic works.
The critical value
The critical value is the threshold value that determines if the calculated value of the test statistic is likely to get if the null-hypothesis is true. It is found in one of the tails of a distribution. If the calculated value exceeds this value (or is less than this value depending on test), the probability is less than the significance level ( α=0.05 or 0.01) that the calculated value belongs to a distribution where the null-hypothesis is true. Then the null-hypothesis is rejected. Otherwise it is accepted and the alternative hypothesis is rejected instead.
- Define H0
- Check that the assumptions for the test holds true
- Calculate the test statistic (e.g. t)
- Compare the calculated value with the critical value that corresponds to the significance level of your choice (mostly α = 0.05 and 0.01). These values are found in tables.
- Reject H0 or H1