Derivation of optimal spacing

Expanding the negative-log-likelihood to the fourth order results in

$$\begin{split} -\log_e f(x\vert\theta^\prime) &= -\log_e f(x\... ...al^4}{\partial \theta^4 } (-\log_e f(x\vert\theta)) \end{split}$$ (33)

The expectation of the zeroth, first and second order terms are as in subsection . The expectation value of the third order term $(\theta^\prime-\theta)^3$ is zero, as it is an odd function (of $\theta^\prime$) about $\theta$. The expectation value of the fourth order is

$$\frac{1}{s(\theta)}\int_{\theta-\frac{s(\theta)}{2}}^{\theta... ...frac{\partial^4}{\partial \theta^4 } (-\log_e f(x\vert\theta))$$ (34)

Since the second-order () and fourth-order terms () are slightly complex, the expressions are simplified via variable substitutions.

$$\begin{split} \textrm{Let~~} F(x,\theta) = \frac{\partial^2}... ...al^4}{\partial \theta^4 } (-\log_e f(x\vert\theta)) \end{split}$$ (35)

This results in the fourth order Taylor series approximation of the Message Length as

$$\textrm{MesgLen~} = -\log_e (h(\theta)s(\theta)) - \log_e f(... ...eta)^2}{24} F(x,\theta) + \frac{s(\theta)^4}{1920} G(x,\theta)$$ (36)

The optimal spacing is derived by finding the value of $s(\theta)$ that minimises the message length. For simplicity, a variable substitution of $z(\theta) = s(\theta)^2$ has then been made. Furthermore, to ensure that the decoder can understand the message, (subsection ), $F(x,\theta)$ and $G(x,\theta)$ were replaced by their respective expectation values.

$$z(\theta) = \frac{-20F(\theta) + \sqrt{(20F(\theta))^2+480G(\theta)}}{G(\theta)}$$ (37)

The positive solution of the quadratic solution has been taken, as it is assumed that $F(\theta) > 0$ and $G(\theta) > 0$. This would generally depend on the ``negative-log-likelihood'' function and $z(\theta)$ might need to be altered to provide a realistic solution. Therefore, the fourth order approximation for the message length is

$$\textrm{MesgLen~} = -\log_e h(\theta) -\frac{1}{2}\log_e z(\... ...theta)}{24} F(x,\theta) + \frac{z(\theta)^2}{1920} G(x,\theta)$$ (38)

In subsection , the assumption that $F(x,\theta) \approx F(\theta)$ was made to preserve invariance. In the fourth order case, even if the assumptions $F(x,\theta) \approx F(\theta)$ and its fourth order equivalent $G(x, \theta) \approx G(\theta)$ were made, the message length expression still would not be invariant. However, numerical experiments have shown that these assumptions are required in order to produce realistic results. Therefore, the fourth order message length expression is simplified to -

$$\textrm{MesgLen~} = -\log_e h(\theta) -\frac{1}{2}\log_e z(\... ...{z(\theta)}{24} F(\theta) + \frac{z(\theta)^2}{1920} G(\theta)$$ (39)

At this stage, it is useful to check that in the case when $G(\theta) = 0$, our fourth order approximation reverts back to the second order approximation. Obviously, equation () corresponds to the second order case when $G(\theta) = 0$ - remembering that $z(\theta) = s(\theta)^2$. However, naively, $z(\theta) = \frac{0}{0}$when $G(\theta) = 0$. In such a case, l'Hospital's rule is required to find the answer.

$$\begin{split} \lim_{G(\theta) \rightarrow 0} z(\theta) &= \li... ...l G(\theta)} G(\theta)} \\ &= \frac{12}{F(\theta)} \end{split}$$ (40)

This provides the same answer as the second order approximation. Albrecht [#!Albrecht:private!#] has mentioned that l'Hospital's rule assumes that the numerator and denominator are independent variables, whereas the above limit was taken in function-space. Therefore, further proof are required to place the work on a sound mathematical basis.