Type I and II error
Everytime you test a null hypothesis using a statistical test you either accept or reject the null hypothesis. In most cases the decision is correct, but we are dealing with probabilities here! You will always reject the null hypothesis if the probability that the observed test statistic is less than the chosen significance level (mostly α = 0.05). There is a chance that the observed value actually belongs to the distribution (of for example no difference between means). The probability that we observe a test statistic that causes us to reject the nullhypothesis when we should not is equal to the chosen significance level.
So, In every test, there is a chance that we reject the null-hypothesis when it should have been accepted. This is called the type I error and have the probability equal to the significance level (α). Similarly, when the nullhypotesis is truly false we might get an observation of the test statistic due to sampling error (pure chance!) that do not cause us to reject the nullhypothesis. This is called the type II error: Acceptance of the nullhypothesis when it should have been rejected. There are four possible outcomes: