Entropy Concentration and the Empirical Coding Game

Simple item page
dc.creatorGrunwald, Peter
dc.date2008-09-05
dc.date.accessioned2026-07-07T10:03:11Z
dc.date.available2026-07-07T10:03:11Z
dc.descriptionWe give a characterization of Maximum Entropy/Minimum Relative Entropy inference by providing two `strong entropy concentration' theorems. These theorems unify and generalize Jaynes' `concentration phenomenon' and Van Campenhout and Cover's `conditional limit theorem'. The theorems characterize exactly in what sense a prior distribution Q conditioned on a given constraint, and the distribution P, minimizing the relative entropy D(P ||Q) over all distributions satisfying the constraint, are `close' to each other. We then apply our theorems to establish the relationship between entropy concentration and a game-theoretic characterization of Maximum Entropy Inference due to Topsoe and others.
dc.descriptionA somewhat modified version of this paper was published in Statistica Neerlandica 62(3), pages 374-392, 2008
dc.identifierhttps://arxiv.org/abs/0809.1017
dc.identifierhttp://arxiv.org/abs/0809.1017
dc.identifier.urihttp://salesiana.dossiersoluciones.com/handle/123456789/169193
dc.subjectInformation Theory
dc.subjectMachine Learning
dc.subjectStatistics Theory
dc.subjectMethodology
dc.titleEntropy Concentration and the Empirical Coding Game
dc.typetext

Files

Collections

arXiv