Root-Mean-Square Distance

The Root-Mean-Square distance is a commonly used scientific measure. It is a measure of the average squared error between a set of parameters.

$$\textrm{RMS}(\theta, \hat{\theta}) = \sqrt{\frac{1}{N} \sum_i^N (\theta_i-\hat{\theta_i})^2}$$ (60)

For the binomial distribution and univariate distributions in general, it simplifies to the absolute error between the estimates.

$$\textrm{RMS}(\theta, \hat{\theta}) = \vert\theta_i-\hat{\theta_i}\vert$$ (61)

  

Figure: Expected Kullback-Leibler distance conditional upon $\theta$ for the binomial distribution

In figure , the parameter $\theta$ is plotted against the ``Expected Kullback-Leibler distance conditional upon $\theta$'' when N=30 or $\sum_{m=0}^{N=30} f(m\vert\theta) \textrm{KL}(\theta, \hat{\theta}(m))$.

The graph is symmetric about $\theta=0.5$ and clearly shows ``4th'' and ``MMLD'' performing better than ``WF87'', but not as well as ``WF87obs'' or ``MEKLD''. For other values of N, the distinctive features of the graph are the same, except that the whole graph scales down to zero as Nincreases.

At the edge of the parameter domain, it is interesting that ``MEKLD'', ``MMLD'' and ``4th'' all decrease sharply, before rising as it approaches the boundary of the domain. ``WF87'' rises slightly, before following the same behaviour, while ``WF87obs'' rises steadily, before dipping then rising again.

  

Figure: Expected Kullback-Leibler distance for the binomial distribution

Figure shows the Expected Kullback-Leibler distance (EKL) for various values of N. It is derived from figure by integrating under the curve over all $\theta$. Mathematically, it is represented by

$$\textrm{EKL}(N) = \int_0^1 \sum_{m=0}^{N} f(m\vert\theta) \textrm{KL}(\theta, \hat{\theta}(m)) d\theta$$ (62)

As the figure shows, N has only varied from 1 to 20, yet it is already difficult to distinguish between the different estimators. As described for Figure , as N increases, the Kullback-Leibler distance and Expected Kullback-Leibler distance decreases to zero.

  

Figure: Relative Expected Kullback-Leibler distance for the binomial distribution

Figure shows the ``Relative Expected Kullback-Leibler distance'' (REKL) for the each of the estimates compared with ``WF87''. Mathematically -

$$\textrm{REKL}(N) = \frac{\textrm{EKL}(N)}{\textrm{EKL}_{WF87}... ...rt\theta) \textrm{KL}(\theta, \hat{\theta}_{WF87}(m)) d\theta}$$ (63)

While the Relative Expected Kullback-Leibler distance affects the apparent distance between estimators, it maintains the relative ordering of the Expected Kullback-Leibler distance. The choice of ``WF87'' was made because it is the basis of this thesis and the MML estimator we are trying to improve.

For small N's (eg N<50), both ``4th'' and ``MMLD'' perform consistently close to the optimal ``MEKLD''. ``WF87obs'' fluctuates when N is small (N<10), but rapidly improves its behaviour, overtaking ``4th'' around N=40. It is clear that both ``4th'' and ``MMLD'' perform better than ``WF87''. As expected, ``MEKLD'' performed the best throughout. As N increases, the results converge to the Maximum Likelihood theory as expected. The fact that ``MMLD'' and ``4th'' does not perform as well as``MEKLD'' does not necessarily detract from their inference abilities. It merely means that their predictive accuracy is not as good as ``MEKLD''.

  

Figure: Expected Root-Mean-Square distance conditional upon $\theta$ for the binomial distribution

Figure shows the ``Expected Root-Mean-Square distance conditional upon $\theta$'' when N=30. Mathematically, this is $\sum_{m=0}^{N=30} f(m\vert\theta) \textrm{RMS}(\theta, \hat{\theta}(m))$. Like figure , the graph is symmetric and it is difficult to see which estimator works best as it varies along $\theta$.

Interestingly, the graph for each of the estimator consists of a series of ``bumps''. Furthermore, as N increases, the number of bumps increases linearly (Appendix ). Not including the half-bumps at the edge of the domain, there are exactly N bumps.

After discussions with Wallace [#!Wallace:private!#], the suggestion that the troughs between the ``bumps'' corresponds with the estimated parameter was verified for ``MEKLD'' and ``WF87''. That is, the troughs occurred at $\theta = \frac{m+1}{N+2}$ for m=[0, N] with ``MEKLD''. Similarly, it occurred at $\theta = \frac{m+\frac{1}{2}}{N+1}$ for m=[0, N] when using ``WF87''. This result have not been verified for the other estimators.

  

Figure: Expected Root-Mean-Square distance for the binomial distribution

Figure shows the Expected RMS distance as a function of N. This is represented mathematically as -

$$\textrm{ERMS}(N) = \int_0^1 \sum_{m=0}^{N} f(m\vert\theta) \textrm{RMS}(\theta, \hat{\theta}(m)) d\theta$$ (64)

On this figure, it is even harder to distinguish between the different estimators than figure with the Kullback-Leibler distance. Again, N has only been taken up to 20. However, it is reassuring that as N increases, the RMS distance decreases to zero as expected.

  

Figure: Relative Expected Root-Mean-Square distance for the binomial distribution

Using the same technique as in figure , figure graphs the Expected RMS distance relative to the ``WF87'' estimator. The definition of the Relative Expected Root-Mean-Square distance is

$$\textrm{RERMS}(N) = \frac{\textrm{ERMS}(N)}{\textrm{ERMS}_{WF... ...t\theta) \textrm{RMS}(\theta, \hat{\theta}_{WF87}(m)) d\theta}$$ (65)

This graph shows more clearly that ``4th'' is a excellent and consistent performer, being beaten by ``MMLD'' only at $N \approx 125$. The behaviour of ``MMLD'' is interesting and mysterious. It fluctuates quite violently when N<10, but improves. Both ``4th'' and ``MMLD'' generally produce better results than ``WF87''. On the other hand, ``MEKLD'' and ``WF87obs'' consistently performs worse than ``WF87''.

There appears to be an numerical artifact at N=99 and N=198. I'm not sure whether it is a coincidence, or whether it will occur every multiple of 99. It suspect it is due to insufficient numerical accuracy when calculating the original data present in figure . This problem is amplified when calculating the relative Root-Mean-Square distance.