Analysis of Variance

Basic Principles

The simplest case of ANOVA is the one-way ANOVA where the groups are varied across only one factor and each group has the same sample size. Suppose there are k groups and within each group there are n samples taken from each group for a total sample size of nk. For notation, let yij be the measurement of outcome of interest from the jth sample in the ith group. We let µi denote the population mean of group i. In this notation, the ANOVA null hypothesis is:

$${\text{H}}_0:{\mu }_1={\mu }_2=\text{L}={\mu }_k.$$

This hypothesis corresponds to the state where all of the means µi are equal to each other and hence do not differ. The alternative hypothesis in this case is:

$${\text{H}}_A:\text{at least two }{\mu }_i\text{differ}.$$

If the ANOVA test rejects H0 in favor of HA, this means there is enough evidence to conclude that the group means are truly different.

To accomplish this, ANOVA partitions the overall variance. The overall variance is simply the sample standard deviation squared of all of the data regardless of treatment group. In our notation, we would have:

$$S^2=\frac{{\displaystyle \sum _{i=1}^k{\displaystyle \sum _{j=1}^n{(y_{ij}-{\overline{y}}_{••})}^2}}}{nk-1},$$

where ${\overline{y}}_{••}=\frac{1}{nk}{\displaystyle \sum _{i=1}^k{\displaystyle \sum _{j=1}^n{y}_{ij}}}$ is the overall mean. In our notation, ${\overline{y}}_{i•}=\frac{1}{n}{\displaystyle \sum _{j=1}^n{y}_{ij}}$ is the sample mean for the ith group. Here, the denominator is not useful in partitioning the groups and will be discarded to create the sum of squares total, denoted by SSTO and is given by:

$$S{S}_{TO}={\displaystyle \sum _{i=1}^k{\displaystyle \sum _{j=1}^n{(y_{ij}-{\overline{y}}_{••})}^2}}$$

By simply adding and subtracting the group sample mean in the SSTO and doing some algebra (some algebra details have been omitted), one can obtain:

$$\begin{array}{l}S{S}_{TO}={\displaystyle \sum _{i=1}^k{\displaystyle \sum _{j=1}^n{(y_{ij}-{\overline{y}}_{i•}+{\overline{y}}_{i•}-{\overline{y}}_{••})}^2}}\\ ={\displaystyle \sum _{i=1}^k{\displaystyle \sum _{j=1}^n[{(y_{ij}-{\overline{y}}_{i•})}^2+2(y_{ij}-{\overline{y}}_{i•})({\overline{y}}_{i•}-{\overline{y}}_{••})}}\\ +{({\overline{y}}_{i•}-{\overline{y}}_{••})}^2]\\ =\dots \\ =\underset{\text{Error}}{\underbrace{{\displaystyle \sum _{i=1}^k{\displaystyle \sum _{j=1}^n{(y_{ij}-{\overline{y}}_{i•})}^2}}}}+\underset{\text{Treatment}}{\underbrace{{\displaystyle \sum _{i=1}^{k}n{({\overline{y}}_{i•}-{\overline{y}}_{••})}^2}}}\\ =S{S}_E+S{S}_T.\end{array}$$

Notice by doing this, the SSTO can be expressed as the sum of a term associated with error and a term associated with the treatment group. This is the essence of ANOVA, partitioning the SSTO into meaningful components. Furthermore, each of the components is itself a sum of items that are squared; hence, the names sum of squares error, SSE, and sum of squares treatment, SST are often given to the components. Note that in this one-way ANOVA scenario, SST is often called the sum of squares between and the SSE is often called the sum of squares within and are denoted by SSB and SSW, respectively.

Similarly, for the one-way ANOVA case with equal sample sizes, the total degrees of freedom, dfTO = nk − 1, associated with the SSTO can also be partitioned into degrees of freedom error, dfE = n(k – 1), and degrees of freedom for treatment, [Page 87]dfT = k – 1. As with the sum of squares, the degrees of freedom also add together nicely dfTO = dfE + dfT.

...