Kullback-Leibler Distance

The Kullback-Leibler distance calculates the coding efficiency between a true probability distribution and an estimated probability distribution. It is not a true distance as it fails the symmetry requirement (ie d(x, y) = d(y, x)).

The Kullback-Leibler distance measures the predictive accuracy of an estimator [#!Farr:1999!#]. In contrast, MML attempts to maximise its ability to infer a theory from a given dataset [#!Farr:1999!#]. Therefore, the Kullback-Leibler distance is not the ideal objective measure for MML estimators, as it measures by the wrong criterion. However, any good inference theory should also be a good (though not perfect) inference theory [#!Farr:1999!#] (and vice versa). As a result, in the absence of a MML objective measure, the Kullback-Leibler distance can be used as an indicator of the accuracy of a MML estimator.

Generally speaking, the Kullback-Leibler distance of an estimated model $\hat{\theta}$ from the true model $\theta$ is given by the expression

$$\textrm{KL}(\theta, \hat{\theta}) = \sum_i^N \theta_i \log_e \frac{\theta_i}{\hat{\theta_i}} \textrm{~nits}$$ (58)

If the probability distribution produces continuous data (as opposed to the 0-state/1-state for the binomial distribution) the summation sign is replaced by an integral.

With the binomial distribution, Kullback-Leibler distance simplifies to -

$$\textrm{KL}(\theta, \hat{\theta}) = \theta \log_e \frac{\the... ...1-\theta)\log_e \frac{1-\theta}{1-\hat{\theta}} \textrm{~nits}$$ (59)