Normal, Gaussian

 MML  KL   M-state   Normal   Normal Also see:   N(,)

KL-distance from Nμ1,σ1 to Nμ2,σ2

General form
x { pdf1(x).{ log(pdf1(x)) - log(pdf2(x)) }}

Two normals, so pdf1(x) is Nμ11(x) etc..

x { Nμ11(x).{ log(Nμ11(x)) - log(Nμ22(x)) }}

```  |                          2         2
|                   1|x-m1|   1|x-m2|      s2
= | {N     (x)} . { - -|----| + -|----| + ln(--)}
|   m1,s1           2| s1 |   2| s2 |      s1
|x
```
Can replace x with x+m1. The expected value of x2 is then s12. Terms that are odd in x, and otherwise symmetric about zero, cancel out over [-∞,∞] leaving the ...x2 and ...constant terms.
```         2       2          2
1|s1|   1|s1|   1|m1-m2|      s2
= - -|--| + -|--| + -|-----| + ln(--)
2|s1|   2|s2|   2| s2  |      s1

2    2    2         2
= {(m1-m2) + s1 - s2 } / (2.s2 ) + ln(s2/s1)
```
This is zero if m1=m2 and s1=s2, it increases with m1-m2 and has rather complex behaviour with s1 and s2  (and is consistent P&R, with KL2 in S,J,R&S, and with J&S where s1=s2).
KL(N(μqq) || N(μpp)), p18 of Penny & Roberts, PARG-00-12, 2000 (see Bib).
Symmetric KL2: KL2(N(μ11), N(μ22)) = (μ12)2. (1/σ12+1/σ22) + σ1222 + σ2212, e.g. Siegler, Jain, Raj, Stern [pdf].
KL(N(μ1,σ), N(μ2,σ)) = (μ12)2/(2σ2), Johnson & Sinanovic, NB. a common σ [pdf].

Note that the distance is convenient to integrate over, say, a range of μ1 & σ1:
 ∫ μ1max ∫ σ1max μ1min σ1min
 (μ1 - μ2)2 2σ22
+ ln σ2 - 1/2  +
 σ12 2σ22
- ln σ1

NB no σ1 here ...

... & no μ1

let  f(&mu1) =
 (μ1 - μ2)3 6σ22
+ μ1 . (ln σ2 - 1/2)
and  g(σ1) =
 σ13 6σ22
- σ1 . (ln σ1 - 1)

= (f(μ1max) - f(μ1min)) . (σ1max - σ1min) + (μ1max - μ1min) . (g(σ1max) - g(σ1min))
window on the wide world:
 The Darwin Awards V: Next Evolution

 Linux  Ubuntu free op. sys. OpenOffice free office suite, ver 3.4+ The GIMP ~ free photoshop Firefox web browser FlashBlock like it says!

 © L. Allison   http://www.allisons.org/ll/   (or as otherwise indicated), Faculty of Information Technology (Clayton), Monash University, Australia 3800 (6/'05 was School of Computer Science and Software Engineering, Fac. Info. Tech., Monash University, was Department of Computer Science, Fac. Comp. & Info. Tech., '89 was Department of Computer Science, Fac. Sci., '68-'71 was Department of Information Science, Fac. Sci.) Created with "vi (Linux + Solaris)",  charset=iso-8859-1,  fetched Friday, 28-Oct-2016 11:35:00 EST.