Automatic divide-and-conquer using populations and ensembles

Summary form only given. Real-world problems are often too large and complex for a single monolithic system to solve. In practice, the divide-and-conquer strategy has often been used to decompose a large and complex problem into smaller tractable sub-problems and then solve them. However, good decomposition of large and complex problems requires experienced human experts and rich prior domain knowledge, which are usually unavailable for real-world problems. This paper explores some of our research efforts towards an adaptive approach to divide-and-conquer in the design of machine learning systems, e.g., evolutionary and neural learning systems. The basic idea is to move away from designing a single monolithic system that would solve a large and complex problem, and to employ a population of simpler sub-systems that will cooperatively solve the problem. In such populations based systems, each sub-system will be simpler and easier to learn than the monolithic system. The integrated system based on the whole population can generalise better than any single subsystems in the population. In particular, by evolving and training a team of specialists from random initial conditions, we were able to "decompose" a large and complex problem into simpler ones and solve them without human intervention (X. Yao et al., 1996). Two major approaches is described. One uses the population structure in evolutionary algorithms, where individuals in a population are evolved into species (i.e., specialists for solving sub-problems) (P. J. Darwen and X. Yao, 1997). The other uses neural network ensembles in which individual neural networks learn to differentiate from and cooperate with each other (Liu and Yao, 1999; and Liu et al., 2000). A constructive algorithm for designing ensembles as well as individual neural networks will be introduced by MM Islam et al. (2003).