Abstract
Negative correlation learning (NCL) is a neural network ensemble learning algorithm that introduces a correlation penalty term to the cost function of each individual network so that each neural network minimizes its mean square error (MSE) together with the correlation of the ensemble. This paper analyzes NCL and reveals that the training of NCL (when λ=1) corresponds to training the entire ensemble as a single learning machine that only minimizes the MSE without regularization. This analysis explains the reason why NCL is prone to overfitting the noise in the training set. This paper also demonstrates that tuning the correlation parameter λ in NCL by cross validation cannot overcome the overfitting problem. The paper analyzes this problem and proposes the regularized negative correlation learning (RNCL) algorithm which incorporates an additional regularization term for the whole ensemble. RNCL decomposes the ensemble's training objectives, including MSE and regularization, into a set of sub-objectives, and each sub-objective is implemented by an individual neural network. In this paper, we also provide a Bayesian interpretation for RNCL and provide an automatic algorithm to optimize regularization parameters based on Bayesian inference. The RNCL formulation is applicable to any nonlinear estimator minimizing the MSE. The experiments on synthetic as well as real-world data sets demonstrate that RNCL achieves better performance than NCL, especially when the noise level is nontrivial in the data set. © 2009 IEEE.
Original language | English |
---|---|
Article number | 5337957 |
Pages (from-to) | 1962-1979 |
Number of pages | 18 |
Journal | IEEE Transactions on Neural Networks |
Volume | 20 |
Issue number | 12 |
Early online date | 23 Nov 2009 |
DOIs | |
Publication status | Published - Dec 2009 |
Externally published | Yes |
Keywords
- Ensembles
- Negative correlation learning (NCL)
- Neural network ensembles
- Neural networks
- Probabilistic model
- Regularization