M7-Quantitative Decision Methods-S4 - Significance Testing

A Significance test aims to make a statement about a population parameter based on sample evidence. It is concerned with the acceptance of rejection of ideas using hypothesis.

In significance testing two types of error can be made:

• A Type I error is the rejection of a null hypothesis when it was actually true. The probability of this error is the significance level of the test.
• A Type II error is the acceptance (or failure to reject) a null hypothesis when it was actually false. The probability of this error cannot be determined.

A good link for further explanation is:

The possible results of a hypothesis test is shown in the table below: ||Accept $H_0$|Reject $H_0$|h |If $H_0$ is correct|Correct decision|Type I error| |If $H_0$ is not correct|Type II error|Correct decision|

Consider the following sample distribution for $\large H_0: \mu = \mu_0$ The two values of the test statistic A and B are both possible with B being less likely than A. The construction of this test with a 5% significance level would mean acceptance of the null hypothesis when A was obtained and the rejection if B was obtained. In reality both values could be attributed to a hypothesis $\large H_1: \mu = \mu_1$ which we accept in the case of B and reject for A. //If the test follows a sample distribution we can never be certain about the correctness of our decision//. It is possible with a fixed significance (Type I error) to construct a rejection region to minimize the type II error. Suppose an alternative hypothesis: $\large H_1: \mu = \mu_1$ where $\large \mu_1 > \mu_0$ or $\large H_1: \mu > \mu_0$ In this case the type II error is as shaded below:

As you can see this type II error possibly has large impact. By testing against an alternative null hypothesis of the form $\large H_1: \mu < \mu_0$ or $\large H_1: \mu > \mu_0$ the size of the type II error can be reduced by carefully defining the rejection region. With

$\large H_1: \mu < \mu_0$ a critical value of z = -1.645 will define the rejection region in the left tail $\large H_1: \mu > \mu_0$ a critical value of z = +1.645 will define the rejection region in the right tail

The reduction in the error is depicted below: