Fisher Information

• Model complexity, -log(H), depends on precision parameter(s) stated to
  
  
• general form based on Fisher information
  
  
• shows sensitivity of -log P(D|H) to estimate
  
  
• high sensitivity: [_________________________]
  
  
• low sensitivity: [_________________________]

Fisher (one parameter)

The 2nd (data) part of message, -ln f(x|theta).

Define:

                  d2
F(theta) = Ex( --------  -ln f(x|theta) )
               d theta2

Prior

The first (estimate) part of message:

h(theta) is prior probability density function of theta.

State theta to ±s/2, assume s small, & h(theta) does not vary much over [theta-s/2, theta+s/2]

Cost to state estimate is:

- ln( h(theta).s ) nits

Rules of the game

• Transmitter sends an estimate theta from the code book
  
  
• estimate represents [theta-s/2, theta+s/2]
  
  
• choose theta to be optimal on average over interval
  
  
• theta' = theta + t, where -s/2<t<s/2

theta' = theta + t, where -s/2<t<s/2

  - ln f(x|theta')

= - ln f(x|theta + t)

                       d
= -ln f(x|theta) +t -------(-ln f(x|theta))
                    d theta

    1      d2
  + - t2 --------(-ln f(x|theta)) + ...
    2    d theta2

Average over [-s/2, s/2],

linear term vanishes

integral of t² over [-s/2,s/2] is s³/12

 

average is:

                 s2    d2
-ln f(x|theta) + --.--------( -ln f(x|theta))
                 24 d theta2

Precision, s

Add two parts of message together:

- ln(h(theta).s) - ln f(x|theta)

  s2    d2
+ --.--------( -ln f(x|theta))
  24 d theta2

s2 = 12 / F(x,theta)

                      d2
where F(x,theta) = -------- -ln( f(x|theta))
                   d theta2

(NB. F(x,theta) is not F(theta), but the two are related...)

S2 = 12/( Ex f(x|theta).F(x,theta) )

   = 12/F(theta)

The Message Length

                               1
- ln h(theta) -ln f(x|theta) + -ln F(theta)
                               2
  1        1 F(x,theta)
- -ln 12 + -.----------
  2        2  F(theta)

                                1
- ln h(theta) - ln f(x|theta) + -ln F(theta)
                                2
                1        1
              - -ln 12 + -
                2        2

Fisher (multiple parameters)

theta = <theta₁, theta₂, ..., theta_n,>

F(x,theta)ij

       d2
= -----------------( -ln f(x|theta))
  d thetai d thetaj


F(theta) = SUMx:X f(x|theta).F(x,theta)

                               1
-ln h(theta) - ln f(x|theta) + -ln F(theta)
                               2

             + -(1 + ln(kn)) nits

Summary

• message length and parameter accuracy
• based on Fisher information.

• h(theta) varies rapidly in region of estimate
• e.g. for small amounts of data

Strict MML (SMML) makes no simplifying assumptions, but may be mathematically and algorithmically difficult.