Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Jicheng LI; S. Joe QIN

doi:10.1016/j.compchemeng.2023.108264

Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Jicheng LI, S. Joe QIN^*

^*Corresponding author for this work

LEO Dr David P. Chan Institute of Data Science

Research output: Journal Publications › Journal Article (refereed) › peer-review

Abstract

Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized.

Original language	English
Article number	108264
Journal	Computers and Chemical Engineering
Volume	175
Early online date	26 Apr 2023
DOIs	https://doi.org/10.1016/j.compchemeng.2023.108264
Publication status	E-pub ahead of print - 26 Apr 2023

Bibliographical note

Funding Information:
Partial financial support for this work is acknowledged from a Natural Science Foundation of China Project ( U20A20189 ), a General Research Fund by the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. 11303421 ), an ITF - Guangdong-Hong Kong Technology Cooperation Funding Scheme (Project Ref. No. GHP/145/20 ), a Math and Application Project ( 2021YFA1003504 ) under the National Key R&D Program, a Shenzhen-Hong Kong-Macau Science and Technology Project Category C ( 9240086 ), an InnoHK initiative of The Government of the HKSAR for the Laboratory for AI-Powered Financial Technologies , and a Collaborative Research Fund (No. C1143-20G ) by RGC of Hong Kong.

Publisher Copyright:
© 2023 Elsevier Ltd

Keywords

Dynamic inferential modeling
LSTM
Partial least squares
Regularized learning
Subspace identification

Access to Document

10.1016/j.compchemeng.2023.108264

Cite this

@article{4b78baaa4944497e86059b5abba8c6f3,

title = "Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling",

abstract = "Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized.",

keywords = "Dynamic inferential modeling, LSTM, Partial least squares, Regularized learning, Subspace identification",

author = "Jicheng LI and QIN, {S. Joe}",

note = "Funding Information: Partial financial support for this work is acknowledged from a Natural Science Foundation of China Project ( U20A20189 ), a General Research Fund by the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. 11303421 ), an ITF - Guangdong-Hong Kong Technology Cooperation Funding Scheme (Project Ref. No. GHP/145/20 ), a Math and Application Project ( 2021YFA1003504 ) under the National Key R&D Program, a Shenzhen-Hong Kong-Macau Science and Technology Project Category C ( 9240086 ), an InnoHK initiative of The Government of the HKSAR for the Laboratory for AI-Powered Financial Technologies , and a Collaborative Research Fund (No. C1143-20G ) by RGC of Hong Kong. Publisher Copyright: {\textcopyright} 2023 Elsevier Ltd",

year = "2023",

month = apr,

day = "26",

doi = "10.1016/j.compchemeng.2023.108264",

language = "English",

volume = "175",

journal = "Computers and Chemical Engineering",

issn = "0098-1354",

publisher = "Elsevier BV",

}

TY - JOUR

T1 - Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

AU - LI, Jicheng

AU - QIN, S. Joe

N1 - Funding Information: Partial financial support for this work is acknowledged from a Natural Science Foundation of China Project ( U20A20189 ), a General Research Fund by the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. 11303421 ), an ITF - Guangdong-Hong Kong Technology Cooperation Funding Scheme (Project Ref. No. GHP/145/20 ), a Math and Application Project ( 2021YFA1003504 ) under the National Key R&D Program, a Shenzhen-Hong Kong-Macau Science and Technology Project Category C ( 9240086 ), an InnoHK initiative of The Government of the HKSAR for the Laboratory for AI-Powered Financial Technologies , and a Collaborative Research Fund (No. C1143-20G ) by RGC of Hong Kong. Publisher Copyright: © 2023 Elsevier Ltd

PY - 2023/4/26

Y1 - 2023/4/26

N2 - Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized.

AB - Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized.

KW - Dynamic inferential modeling

KW - LSTM

KW - Partial least squares

KW - Regularized learning

KW - Subspace identification

UR - http://www.scopus.com/inward/record.url?scp=85154581845&partnerID=8YFLogxK

U2 - 10.1016/j.compchemeng.2023.108264

DO - 10.1016/j.compchemeng.2023.108264

M3 - Journal Article (refereed)

SN - 0098-1354

VL - 175

JO - Computers and Chemical Engineering

JF - Computers and Chemical Engineering

M1 - 108264

ER -

Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this