Regularized negative correlation learning for neural network ensembles

Huanhuan CHEN, Xin YAO

Research output: Journal PublicationsJournal Article (refereed)peer-review

121 Citations (Scopus)

Abstract

Negative correlation learning (NCL) is a neural network ensemble learning algorithm that introduces a correlation penalty term to the cost function of each individual network so that each neural network minimizes its mean square error (MSE) together with the correlation of the ensemble. This paper analyzes NCL and reveals that the training of NCL (when λ=1) corresponds to training the entire ensemble as a single learning machine that only minimizes the MSE without regularization. This analysis explains the reason why NCL is prone to overfitting the noise in the training set. This paper also demonstrates that tuning the correlation parameter λ in NCL by cross validation cannot overcome the overfitting problem. The paper analyzes this problem and proposes the regularized negative correlation learning (RNCL) algorithm which incorporates an additional regularization term for the whole ensemble. RNCL decomposes the ensemble's training objectives, including MSE and regularization, into a set of sub-objectives, and each sub-objective is implemented by an individual neural network. In this paper, we also provide a Bayesian interpretation for RNCL and provide an automatic algorithm to optimize regularization parameters based on Bayesian inference. The RNCL formulation is applicable to any nonlinear estimator minimizing the MSE. The experiments on synthetic as well as real-world data sets demonstrate that RNCL achieves better performance than NCL, especially when the noise level is nontrivial in the data set. © 2009 IEEE.
Original languageEnglish
Article number5337957
Pages (from-to)1962-1979
Number of pages18
JournalIEEE Transactions on Neural Networks
Volume20
Issue number12
Early online date23 Nov 2009
DOIs
Publication statusPublished - Dec 2009
Externally publishedYes

Bibliographical note

The work of H. Chen was supported in part by a Dorothy Hodgkin Postgraduate award. The work of X. Yao was supported by EPSRC under Grant GR/T10671/01.

Keywords

  • Ensembles
  • Negative correlation learning (NCL)
  • Neural network ensembles
  • Neural networks
  • Probabilistic model
  • Regularization

Fingerprint

Dive into the research topics of 'Regularized negative correlation learning for neural network ensembles'. Together they form a unique fingerprint.

Cite this