Information theory and learning: a physical approach
| dc.creator | Nemenman, Ilya | |
| dc.date | 2000-09-08 | |
| dc.date.accessioned | 2026-07-07T05:44:47Z | |
| dc.date.available | 2026-07-07T05:44:47Z | |
| dc.description | We try to establish a unified information theoretic approach to learning and to explore some of its applications. First, we define {\em predictive information} as the mutual information between the past and the future of a time series, discuss its behavior as a function of the length of the series, and explain how other quantities of interest studied previously in learning theory - as well as in dynamical systems and statistical mechanics - emerge from this universally definable concept. We then prove that predictive information provides the {\em unique measure for the complexity} of dynamics underlying the time series and show that there are classes of models characterized by {\em power-law growth of the predictive information} that are qualitatively more complex than any of the systems that have been investigated before. Further, we investigate numerically the learning of a nonparametric probability density, which is an example of a problem with power-law complexity, and show that the proper Bayesian formulation of this problem provides for the `Occam' factors that punish overly complex models and thus allow one {\em to learn not only a solution within a specific model class, but also the class itself} using the data only and with very few a priori assumptions. We study a possible {\em information theoretic method} that regularizes the learning of an undersampled discrete variable, and show that learning in such a setup goes through stages of very different complexities. Finally, we discuss how all of these ideas may be useful in various problems in physics, statistics, and, most importantly, biology. | |
| dc.description | 140 pages; Ph.D. thesis, Princeton University, department of Physics | |
| dc.identifier | https://arxiv.org/abs/physics/0009032 | |
| dc.identifier | http://arxiv.org/abs/physics/0009032 | |
| dc.identifier.uri | http://salesiana.dossiersoluciones.com/handle/123456789/83780 | |
| dc.subject | Data Analysis, Statistics and Probability | |
| dc.subject | Disordered Systems and Neural Networks | |
| dc.subject | Machine Learning | |
| dc.subject | Adaptation and Self-Organizing Systems | |
| dc.title | Information theory and learning: a physical approach | |
| dc.type | text |