Methodology and Results

In this section, five different MML estimators were applied to the binomial distribution (section ) and the results compared using two objective functions. Of the five, MMLD and fourth-order have been developed in this thesis, while the other three have been previously published. As mentioned previously, the asymmetric coding region estimator has not been sufficiently developed for an estimator to be calculated. The five estimators and their descriptions are below.

1.

The estimator found in Wallace and Boulton [#!Wallace.Boulton:1968!#, page 187 eqn 3, page 189 eqn 30]. $\hat{\theta}_{MEKLD} = \frac{m+1}{N+2}$This estimator has been proven to be the Minimum Expected Kullback-Leibler Distance (MEKLD) [#!Dowe.Baxter.Oliver.Wallace:1998!#]. Therefore, using the Kullback-Leibler distance as an objective measure does not give the fairest comparison. However, this estimator came from the original paper that originated the MML principle [#!Wallace.Boulton:1968!#]. It also provides a lower bound for the Expected Kullback-Leibler distance. Furthermore, any estimator which minimises the Expected Kullback-Leibler distance, would also be expected to be a good (though not optimal) MML estimator [#!Farr:1999!#].

2.

Wallace and Freeman's 1987 estimator [#!Wallace.Freeman:1987!#], the basis of this thesis. This estimator is also found implicitly in Wallace and Boulton [#!Wallace.Boulton:1968!#, page 188 eqn 4, page 194 eqn 28]. This second order approximation is invariant and easily calculated. With the binomial distribution, the estimator is $\hat{\theta}_{WF87} = \frac{m+\frac{1}{2}}{N+1}$.

3.

Wallace and Freeman's 1987 estimator with Observed Fisher [#!Wallace:2000!#]. This MML estimator, which is not invariant, has also been labeled the Farr-Wallace estimator [#!Farr:1999!#]. As mentioned in subsection , the assumption that $F(\theta) \approx F(m, \theta)$ was made in order to make the estimator invariant. However, this causes a breakdown in the MML estimator when this approximation fails [#!Grunwald.Kontkanen.Myllymaki.Silander.Tirri:1998!#,#!Grunwald:1998!#]. This estimator is used as a comparison with estimator 2. It is calculated by numerically solving equation () with respect to $\theta$.

4.

Fourth-order extension to Wallace and Freeman's estimator. This improves estimator 2 by extending the Taylor series approximation to the fourth order. It is described in section and more specifically for the binomial distribution in subsection .

5.

MMLD, as described in section and specifically for the binomial distribution in subsection .

For reasons of brevity, ``Minimum Expected Kullback-Leibler Distance'' (estimator 1) will be referred to as ``MEKLD'' and similarly for the other estimators. ``Wallace and Freeman 1987'' (estimator 2) will be referred as ``WF87''. ``Wallace and Freeman 1987 with Observed Fisher'' (estimator 3) will be referred as ``WF87obs''. ``Fourth-order extension to Wallace and Freeman'' (estimator 4) will be referred as ``4th''. ``MMLD'' (estimator 5) will remain ``MMLD''.

To compare the different estimators, two objective measures were used. The first is the Kullback-Leibler distance and the second is the Root-Mean-Square distance. They are described in greater detail below. Appendix also contains a summary of the plotted data in tabular form to a higher precision than could be displayed in the figures.