Kullback Leibler Distance (KL)

LA home
Computing
 Algorithms
 Bioinformatics
 FP,  λ
 Logic,  π
 MML
 Prog.Langs

MML
 Glossary
 Discrete
 Continuous
 Structured
 SMML
 KL-dist
 "Art"
 Ind. Inf.
 KL Distance
  M-state
  Normal
The Kullback Leibler distance (KL-distance) is a natural distance function from a "true" probability distribution, p, to a "target" probability distribution, q. It can be interpreted as the expected extra message-length per datum due to using a code based on the wrong (target) distribution compared to using a code based on the true distribution.
 
For discrete (not necessarily finite) probability distributions, p={p1, ..., pn} and q={q1, ..., qn}, the KL-distance is defined to be
 
KL(p, q) = Σi pi . log2( pi / qi )
 
For continuous probability densities, the sum is replaced by an integral.
 
KL(p, p) = 0
KL(p, q) ≥ 0
 
Note that the KL-distance is not, in general, symmetric.
window on the wide world:

Computer Science Education Week

Linux
 Ubuntu
free op. sys.
OpenOffice
free office suite,
ver 3.4+

The GIMP
~ free photoshop
Firefox
web browser
FlashBlock
like it says!

Also see:
 II
  ACSC06
  JFP05
  ACSC03

© L. Allison   http://www.allisons.org/ll/   (or as otherwise indicated),
Faculty of Information Technology (Clayton), Monash University, Australia 3800 (6/'05 was School of Computer Science and Software Engineering, Fac. Info. Tech., Monash University,
was Department of Computer Science, Fac. Comp. & Info. Tech., '89 was Department of Computer Science, Fac. Sci., '68-'71 was Department of Information Science, Fac. Sci.)
Created with "vi (Linux + Solaris)",  charset=iso-8859-1,  fetched Monday, 21-Apr-2014 17:09:01 EST.