Bagging

Summary

Bagging (a.k.a bootstrap aggregating) is an ensemble machine learning method introduced by Leo Brieman for classification and regression, which generates multiple versions of a predictor, by making bootstrap replications of the learning set and using them as the new learning set, and uses them to produce an aggregated predictor, which does voting over the different versions for classification and averages outcomes when predicting numerical values. Brieman showed that bagging can best improve accuracy when the predictors are good but unstable (when perturbing the learning set results in significant changes in the predictors).

Procedure

Input/Definitions:

• D - Training set
• D_i - One of the bootstrap sample training sets
• n - Size of D
• m - Number of predictors to construct
• M - Set of trained models
• M_i - One of the trained models

Training:

• Generate m new training sets, D_i, of size n_prime < n.
  • Do so by sampling from D uniformly and with replacement (a.k.a. a bootstrap sample)
• Train m models, M_i, using bootstrap sample D_i

Output Prediction:

• For Regression: Average outcome of the predictors in M
• For Classification: Vote of the m predictors

References / Links

• Leo Brieman. Bagging Predictors. Machine Learning, 24, 123–140 (1996).
• Wikipedia article on bagging.