Title:
MML, statistically consistent invariant Bayesian probabilistic inference and
the elusive model paradox
Speaker:
David L. Dowe - Clayton School of Information Technology, Monash University,
Clayton (Melbourne), Australia
[and Departmento de Sistemas Informatics y Computacion, Universidad
Politecnica de Valencia (UPV), Valencia, Espan~a]
(Brief?) speaker bio':
David Dowe is Associate Professor in the Clayton School of Information
Technology (formerly Department of Computer Science) at Monash University in
Melbourne, Australia. He was the closest collaborator of Chris Wallace
(1933-2004), the originator (Wallace and Boulton, Computer J, 1968) of Minimum
Message Length (MML). In Wallace's (posthumous, 2005) book on MML, Dowe is the
most cited and most discussed living person.
Dowe was guest editor of the Christopher Stewart WALLACE (1933-2004) memorial
special issue of the Computer J (Vol. 51, No. 5 [Sept. 2008]) and will be chair
of the Ray Solomonoff (1926-2009) 85th memorial conference in late 2011. Dowe's
many contributions and applications of MML to machine learning, statistics and
philosophy, etc. include (e.g.)
(i) his conjecture that only MML and close Bayesian approximations can - in
general - guarantee both statistical invariance and statistical consistency
(e.g., Dowe, Gardner & Oppy, Brit J Philos Sci, 2007),
(ii) his 2010 book chapter on MML and philosophy of statistics in the Handbook
of the Philosophy of Science - (HPS Volume 7) Philosophy of Statistics,
Elsevier [ISBN: 978-0-444-51862],
(iii) his work (Dowe & Hajek, 1998) (Hernandez Orallo & Dowe, AI journal, 2010)
using MML as an response to Searle's Chinese room argument and showing how MML
can be used to quantify intelligence (http://users.dsic.upv.es/proy/anynt),
(iv) his uniqueness result about the invariance of log-loss probabilistic
scoring,
(v) his supervision of the world's longest running (since 1995)
compression-based log-loss competition (at www.csse.monash.edu.au/~footy),
(vi) the first work on MML Bayesian nets using both discrete and continuous
variables, etc.
Abstract:
We outline the Minimum Message Length (MML) principle (from Bayesian
information theory) (Wallace and Boulton, Computer J, 1968) of statistics,
machine learning, econometrics, inductive inference and (so-called) data mining.
We explain the result from elementary information theory that the information
content (or code length) of an event is the negative logarithm of its
probability, l = - log(p). We also mention the notion of statistical
invariance (if analysing a cube, the estimate of the volume should be the cube
of the estimate of the side length; if analysing a circle, the estimate of the
area should be pi times the square of the estimate of the radius, etc.) and the
notion of statistical consistency (as you get more and more data, your estimate
converges as closely as is possible to the correct answer). As desiderata,
statistical invariance seems more than aesthetic, and statistical consistency
seems much more than aesthetic.
We relate MML to algorithmic information theory (or Kolmogorov complexity)
(Wallace & Dowe, Computer J, 1999a), essentially the amount of information
required to program a (Universal) Turing machine.
We then highlight MML's ability to deal with problems where the amount of data
(per parameter) is scarce (such as, e.g., latent factor analysis of (e.g.) I.Q.
or of octane rating, or the classic Neyman-Scott (1948) problem). It is this
ability of MML which lead to the speaker's conjecture (Dowe et al., 1998;
Dowe, 2010) about the uniqueness of MML in being able to be both statistically
invariant and statistically consistent for problems where the amount of data
per parameter is bounded above.
We then relate MML to a few problems in the philosophy of science.
Depending upon time and the wishes of the audience, etc., such problems might
be (e.g.)
(i) the elusive model paradox (Scriven, 1965; Lewis & Shelby Richardson,
1966; Dowe 2008a, 2008b, 2010),
(ii) ``objective'' (Bayesian) inference, inference (and explanation)
(Wallace & Boulton, 1968) versus prediction (Solomonoff, 1964),
(iii) MML, Searle's Chinese room and ``intelligence'',
(iv) Goodman's ``grue'' problem (paradox) of induction,
(v) inevitability of financial market inefficiency,
(vi) probabilities of conditionals,
(vii) entropy not being the arrow of time,
(viii) fictionalism, etc.