Normal, Gaussian

LA home
Computing
 Algorithms
 Bioinformatics
 FP,  λ
 Logic,  π
 MML
 Prog.Langs

MML
 KL
  M-state
  Normal
  Normal

Also see:
  N(,)

KL-distance from Nμ11 to Nμ22

General form
x { pdf1(x).{ log(pdf1(x)) - log(pdf2(x)) }}
 
Two normals, so pdf1(x) is Nμ11(x) etc..
 
x { Nμ11(x).{ log(Nμ11(x)) - log(Nμ22(x)) }}
 
  |                          2         2
  |                   1|x-m1|   1|x-m2|      s2
= | {N     (x)} . { - -|----| + -|----| + ln(--)}
  |   m1,s1           2| s1 |   2| s2 |      s1
  |x
Can replace x with x+m1. The expected value of x2 is then s12. Terms that are odd in x, and otherwise symmetric about zero, cancel out over [-∞,∞] leaving the ...x2 and ...constant terms.
         2       2          2
    1|s1|   1|s1|   1|m1-m2|      s2
= - -|--| + -|--| + -|-----| + ln(--)
    2|s1|   2|s2|   2| s2  |      s1

          2    2    2         2
= {(m1-m2) + s1 - s2 } / (2.s2 ) + ln(s2/s1)
This is zero if m1=m2 and s1=s2, it increases with m1-m2 and has rather complex behaviour with s1 and s2  (and is consistent P&R, with KL2 in S,J,R&S, and with J&S where s1=s2).
KL(N(μqq) || N(μpp)), p18 of Penny & Roberts, PARG-00-12, 2000 (see Bib).
Symmetric KL2: KL2(N(μ11), N(μ22)) = (μ12)2. (1/σ12+1/σ22) + σ1222 + σ2212, e.g. Siegler, Jain, Raj, Stern [pdf].
KL(N(μ1,σ), N(μ2,σ)) = (μ12)2/(2σ2), Johnson & Sinanovic, NB. a common σ [pdf].

 
Note that the distance is convenient to integrate over, say, a range of μ1 & σ1:
μ1max σ1max   
   
μ1min σ1min
1 - μ2)2
22
+ ln σ2 - 1/2  + 
σ12
22
 - ln σ1
 
NB no σ1 here ...
 
... & no μ1
 
let  f(&mu1) =
1 - μ2)3
22
+ μ1 . (ln σ2 - 1/2)
and  g(σ1) =
σ13
22
  - σ1 . (ln σ1 - 1)
 
= (f(μ1max) - f(μ1min)) . (σ1max - σ1min) + (μ1max - μ1min) . (g(σ1max) - g(σ1min))
window on the wide world:

Computer Science Education Week

Linux
 Ubuntu
free op. sys.
OpenOffice
free office suite,
ver 3.4+

The GIMP
~ free photoshop
Firefox
web browser
FlashBlock
like it says!

© L. Allison   http://www.allisons.org/ll/   (or as otherwise indicated),
Faculty of Information Technology (Clayton), Monash University, Australia 3800 (6/'05 was School of Computer Science and Software Engineering, Fac. Info. Tech., Monash University,
was Department of Computer Science, Fac. Comp. & Info. Tech., '89 was Department of Computer Science, Fac. Sci., '68-'71 was Department of Information Science, Fac. Sci.)
Created with "vi (Linux + Solaris)",  charset=iso-8859-1,  fetched Monday, 29-Dec-2014 14:12:44 EST.