Applying Exponential Family Distribution to Generalized Extreme Learning Machine

Yuheng JIA, Sam KWONG, Ran WANG

Research output: Journal PublicationsJournal Article (refereed)peer-review

4 Citations (Scopus)


The learning algorithm of an extreme learning machine (ELM) has two fundamental steps: 1) random nonlinear feature transformation and 2) least squares learning. Since the probabilistic interpretation for a sample by the least squares method follows a Gaussian distribution, there are two limitations in ELM caused by the second step: 1) it may be inaccurate to handle binary classification problems, since the output of a binary dataset has a distribution far from Gaussian and 2) it may have difficulties in dealing with nontraditional data types (such as count data, ordinal data, etc.), which also do not follow Gaussian distribution. In order to solve the above-mentioned problems, this paper proposes a generalized ELM (GELM) framework by applying the exponential family distribution (EFD) to the output layer node of ELM. It simplifies the design of ELM models for task-specific output domains with different data types. We propose a unified learning paradigm for all the models under this GELM framework with different distributions in EFD, and prove that traditional ELM is a special instance of GELM by setting the output distribution as a Gaussian distribution (GELM-Gaussian). We also prove that the training of GELM-Gaussian can be finished in one iteration, in this case, GELM-Gaussian does not slow down the training speed of traditional ELM. Besides, we propose the kernel version of GELM, which can also be concretized to different models by applying different EFDs. Experimental comparisons demonstrate that GELM can give more accurate probabilistic interpretation to binary classification and GELM has a great potential in dealing with a broader range of machine learning tasks.
Original languageEnglish
Pages (from-to)1794-1804
JournalIEEE Transactions on Systems, Man, and Cybernetics: Systems
Issue number5
Early online date23 Jan 2018
Publication statusPublished - May 2020
Externally publishedYes

Bibliographical note

This work was supported in part by the Natural Science Foundation of China under Grant 61672443, Grant 61772344, and Grant 61732011, in part by the Hong Kong RGC General Research Funds (9042489) under Grant CityU 11206317 and the Hong Kong RGC General Research Funds (9042322) under Grant CityU 11200116, and in part by the Natural Science Foundation of SZU under Grant 827-000230.


  • Exponential family distribution (EFD)
  • generalized extreme learning machine (GELM)
  • kernel generalized extreme learning machine (KGELM)
  • likelihood function
  • probabilistic interpretation


Dive into the research topics of 'Applying Exponential Family Distribution to Generalized Extreme Learning Machine'. Together they form a unique fingerprint.

Cite this