A Bernstein-Von Mises Theorem for discrete probability distributions
Abstract
Description
We investigate the asymptotic normality of the posterior distribution in the discrete setting, when model dimension increases with sample size. We consider a probability mass function $θ_0$ on $\mathbbm{N}\setminus \{0\}$ and a sequence of truncation levels $(k_n)_n$ satisfying $k_n^3\leq n\inf_{i\leq k_n}θ_0(i).$ Let $\hatθ$ denote the maximum likelihood estimate of $(θ_0(i))_{i\leq k_n}$ and let $Δ_n(θ_0)$ denote the $k_n$-dimensional vector which $i$-th coordinate is defined by \sqrt{n} (\hatθ_n(i)-θ_0(i)) for $1\leq i\leq k_n.$ We check that under mild conditions on $θ_0$ and on the sequence of prior probabilities on the $k_n$-dimensional simplices, after centering and rescaling, the variation distance between the posterior distribution recentered around $\hatθ_n$ and rescaled by $\sqrt{n}$ and the $k_n$-dimensional Gaussian distribution $\mathcal{N}(Δ_n(θ_0),I^{-1}(θ_0))$ converges in probability to $0.$ This theorem can be used to prove the asymptotic normality of Bayesian estimators of Shannon and Rényi entropies. The proofs are based on concentration inequalities for centered and non-centered Chi-square (Pearson) statistics. The latter allow to establish posterior concentration rates with respect to Fisher distance rather than with respect to the Hellinger distance as it is commonplace in non-parametric Bayesian statistics.
Published in at http://dx.doi.org/10.1214/08-EJS262 the Electronic Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of Mathematical Statistics (http://www.imstat.org)
Published in at http://dx.doi.org/10.1214/08-EJS262 the Electronic Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of Mathematical Statistics (http://www.imstat.org)