The procedure of statistical tests

Before a sta­tis­ti­cal test is con­duct­ed, the alter­na­tive (H1) and null-hypoth­e­sis (H0) should be clear­ly defined as argued in the hypoth­e­sis test­ing sec­tion.  The pro­ce­dure of a sta­tis­ti­cal test and the result­ing rejec­tion of either H0 or H1 fol­lows a gen­er­al pro­ce­dure where a test sta­tis­tic is cal­cu­lat­ed which is then com­pared to a crit­i­cal thresh­old val­ue. If the test sta­tis­tic is larg­er or small­er than the crit­i­cal val­ue of the sta­tis­tic, depend­ing on the test, H0 or H1 is reject­ed.  The rejec­tion of H0 pro­vides sup­port for H1 and vice ver­ca.

Check assumptions

Each test relies on a set of assump­tions. In the t test for exam­ple, the sam­ples involved needs to be nor­mal­ly dis­trib­uted, have equal vari­ances and obser­va­tions must be inde­pen­dent. The results of the test is not reli­able if the assump­tions are vio­lat­ed. If they are, anoth­er test may be more appro­pri­ate.  Para­met­ric tests (e.g. the Z-test, t-test and ANOVA) are more bound­ed by spe­cif­ic assump­tions than non-para­met­ric tests (e.g. Mann-Whit­ney, Spear­man and Kruskal Wal­lis test). The lat­ter only relies on the assump­tion of inde­pen­dence. How­ev­er, these tests have low­er pow­er com­pared to para­met­ric tests. So, what you need to do before you pro­ceed with a test is to check the assump­tions.

The assump­tion of nor­mal­i­ty is eas­i­ly done by a his­togram, which show if the sam­ple at hand fol­lows a nor­mal dis­tri­b­u­tion. It does not has to be per­fect. If the sam­ple size is small, this assump­tion may be dif­fi­cult to test. Don’t wor­ry. Sim­u­la­tions have shown that para­met­ric test in fact are quite robust to vio­la­tions of this assump­tion.  A worse sce­nario is though that vari­ances are unequal. This assump­tion is checked by a F-test.

The test statistic

Every test sta­tis­tic is part of a math­e­mat­i­cal­ly defined dis­tri­b­u­tion. The cal­cu­lat­ed val­ue of a test sta­tis­tic express­es its posi­tion with­in a dis­tri­b­u­tion of pos­si­ble val­ues of the sta­tis­tic if the null-hypoth­e­sis is true. If the test sta­tis­tic is not like­ly to get with­in this dis­tri­b­u­tion, the null-hypoth­e­sis is reject­ed.  Which val­ues that are “unlike­ly” is set by the sig­nif­i­cance lev­el of the test (α=0.05 or 0.01). If the prob­a­bil­i­ty of get­ting the cal­cu­lat­ed val­ue of the test sta­tis­tic if the null-hypoth­e­sis holds true is less than for exam­ple  0.05, the null-hypoth­e­sis is reject­ed. See the Z-test sec­tion for a detailed descrip­tion of how the Z test sta­tis­tic works.

The critical value

The crit­i­cal val­ue is the thresh­old val­ue that deter­mines if the cal­cu­lat­ed val­ue of the test sta­tis­tic is like­ly to get if the null-hypoth­e­sis is true. It is found in one of the tails of a dis­tri­b­u­tion. If the cal­cu­lat­ed val­ue exceeds this val­ue (or is less than this val­ue depend­ing on test), the prob­a­bil­i­ty is less than the sig­nif­i­cance lev­el ( α=0.05 or 0.01) that the cal­cu­lat­ed val­ue belongs to a dis­tri­b­u­tion where the null-hypoth­e­sis is true. Then the null-hypoth­e­sis is reject­ed. Oth­er­wise it is accept­ed and the alter­na­tive hypoth­e­sis is reject­ed instead.

Summary

  1. Define H0
  2. Check that the assump­tions for the test holds true
  3. Cal­cu­late the test sta­tis­tic (e.g. t)
  4. Com­pare the cal­cu­lat­ed val­ue with the crit­i­cal val­ue that cor­re­sponds to the sig­nif­i­cance lev­el of your choice (most­ly α = 0.05 and 0.01). These val­ues are found in tables.
  5. Reject H0 or  H1