M7-Quantitative Decision Methods-S4 - Significance Testing

A Significance test aims to make a statement about a population parameter based on sample evidence. It is concerned with the acceptance of rejection of ideas using hypothesis.

In significance testing two types of error can be made:

• A Type I error is the rejection of a null hypothesis when it was actually true. The probability of this error is the significance level of the test.
• A Type II error is the acceptance (or failure to reject) a null hypothesis when it was actually false. The probability of this error cannot be determined.

A good link for further explanation is:

The possible results of a hypothesis test is shown in the table below:

	Accept $H_{0}$	Reject $H_{0}$
If $H_{0}$ is correct	Correct decision	Type I error
If $H_{0}$ is not correct	Type II error	Correct decision

Consider the following sample distribution for $H_{0}:\mu =\mu _{0}$ The two values of the test statistic A and B are both possible with B being less likely than A. The construction of this test with a 5% significance level would mean acceptance of the null hypothesis when A was obtained and the rejection if B was obtained. In reality both values could be attributed to a hypothesis $H_{1}:\mu =\mu _{1}$ which we accept in the case of B and reject for A. If the test follows a sample distribution we can never be certain about the correctness of our decision. It is possible with a fixed significance (Type I error) to construct a rejection region to minimize the type II error. Suppose an alternative hypothesis: $H_{1}:\mu =\mu _{1}$ where $\mu _{1}>\mu _{0}$ or $H_{1}:\mu >\mu _{0}$ In this case the type II error is as shaded below:

As you can see this type II error possibly has large impact. By testing against an alternative null hypothesis of the form $H_{1}:\mu <\mu _{0}$ or $H_{1}:\mu >\mu _{0}$ the size of the type II error can be reduced by carefully defining the rejection region. With

$H_{1}:\mu <\mu _{0}$ a critical value of z = -1.645 will define the rejection region in the left tail $H_{1}:\mu >\mu _{0}$ a critical value of z = +1.645 will define the rejection region in the right tail

The reduction in the error is depicted below: