Bounding the generalization error of convex combinations of classifiers: balancing the dimensionality and the margins

Koltchinskii, Vladimir; Panchenko, Dmitry; Lozano, Fernando

Bounding the generalization error of convex combinations of classifiers: balancing the dimensionality and the margins

Authors

Koltchinskii, Vladimir

Panchenko, Dmitry

Lozano, Fernando

Description

A problem of bounding the generalization error of a classifier f in H, where H is a "base" class of functions (classifiers), is considered. This problem frequently occurs in computer learning, where efficient algorithms of combining simple classifiers into a complex one (such as boosting and bagging) have attracted a lot of attention. Using Talagrand's concentration inequalities for empirical processes, we obtain new sharper bounds on the generalization error of combined classifiers that take into account both the empirical distribution of "classification margins'' and an "approximate dimension" of the classifiers and study the performance of these bounds in several experiments with learning algorithms.
35 pages, 7 figures