Predictive ensemble pruning by expectation propagation

Huanhuan CHEN, Peter TIŇO, Xin YAO

Research output: Journal PublicationsJournal Article (refereed)peer-review

96 Citations (Scopus)

Abstract

An ensemble is a group of learners that work together as a committee to solve a problem. The existing ensemble learning algorithms often generate unnecessarily large ensembles, which consume extra computational resource and may degrade the generalization performance. Ensemble pruning algorithms aim to find a good subset of ensemble members to constitute a small ensemble, which saves the computational resource and performs as well as, or better than, the unpruned ensemble. This paper introduces a probabilistic ensemble pruning algorithm by choosing a set of "sparse" combination weights, most of which are zeros, to prune the ensemble. In order to obtain the set of sparse combination weights and satisfy the nonnegative constraint of the combination weights, a left-truncated, nonnegative, Gaussian prior is adopted over every combination weight. Expectation propagation (EP) algorithm is employed to approximate the posterior estimation of the weight vector. The leave-one-out (LOO) error can be obtained as a by-product in the training of EP without extra computation and is a good indication for the generalization error. Therefore, the LOO error is used together with the Bayesian evidence for model selection in this algorithm. An empirical study on several regression and classification benchmark data sets shows that our algorithm utilizes far less component learners but performs as well as, or better than, the unpruned ensemble. Our results are very competitive compared with other ensemble pruning algorithms. © 2009 IEEE.
Original languageEnglish
Article number4798164
Pages (from-to)999-1013
Number of pages15
JournalIEEE Transactions on Knowledge and Data Engineering
Volume21
Issue number7
Early online date10 Mar 2009
DOIs
Publication statusPublished - Jul 2009
Externally publishedYes

Funding

This work was partially supported by a Dorothy Hodgkin Postgraduate Award (DHPA) to Huanhuan Chen.

Keywords

  • Classification
  • Ensemble learning
  • Machine learning
  • Probabilistic algorithms
  • Regression

Fingerprint

Dive into the research topics of 'Predictive ensemble pruning by expectation propagation'. Together they form a unique fingerprint.

Cite this