Surrogate-Assisted Evolutionary Q-Learning for Black-Box Dynamic Time-Linkage Optimization Problems

Tuo ZHANG, Handing WANG, Bo YUAN, Yaochu JIN, Xin YAO

Research output: Journal PublicationsJournal Article (refereed)peer-review

7 Citations (Scopus)


Dynamic time-linkage optimization problems (DTPs) are special dynamic optimization problems (DOPs) with the time-linkage property. The environment of DTPs changes not only over time but also depends on the previous applied solutions. DTPs are hardly solved by existing dynamic evolutionary algorithms because they ignore the time-linkage property. In fact, they can be viewed as multiple decision-making problems and solved by reinforcement learning (RL). However, only some discrete DTPs are solved by RL-based evolutionary optimization algorithms with the assumption of observable objective functions. In this work, we propose a dynamic evolutionary optimization algorithm using surrogate-assisted Q-learning for continuous black-box DTPs. To observe the states of black-box DTPs, the state extraction and prediction methods are applied after the search process at each time step. Based on the learned information, a surrogate-assisted Q-learning is introduced to evaluate and select candidate solutions in the continuous decision space in a long-term consideration. We evaluate the components of our proposed algorithm on various benchmark problems to study their behaviors. Results of comparative experiments indicate that the proposed algorithm outperforms other compared algorithms and performs robustly on DTPs with up to 30 decision variables and different dynamic changes. © 1997-2012 IEEE.
Original languageEnglish
Pages (from-to)1162-1176
Number of pages15
JournalIEEE Transactions on Evolutionary Computation
Issue number5
Early online date30 May 2022
Publication statusPublished - Oct 2023
Externally publishedYes

Bibliographical note

This work was supported in part by the National Natural Science Foundation of China under Grant 61976165, and in part by the Guangdong Provincial Key Laboratory under Grant 2020B121201001. The work of Yaochu Jin was supported by the Alexander von Humboldt Professorship for Artificial Intelligence funded by the German Federal Ministry of Education and Research.


  • Black-box problem
  • dynamic time-linkage optimization problem (DTP)
  • evolutionary dynamic optimization (EDO)
  • Q-learning
  • surrogate model


Dive into the research topics of 'Surrogate-Assisted Evolutionary Q-Learning for Black-Box Dynamic Time-Linkage Optimization Problems'. Together they form a unique fingerprint.

Cite this