HELP : An LSTM-based approach to hyperparameter exploration in neural network learning

Wendi LI, Wing W. Y. NG, Ting WANG, Marcello PELILLO, Sam KWONG

Research output: Journal PublicationsJournal Article (refereed)peer-review

25 Citations (Scopus)

Abstract

Hyperparameter selection is very important for the success of deep neural network training. Random search of hyperparameters for deep neural networks may take a long time to converge and yield good results because the training of deep neural networks with a huge number of parameters for every selected hyperparameter is very time-consuming. In this work, we propose the Hyperparameter Exploration LSTM-Predictor (HELP) which is an improved random exploring method using a probability-based exploration with an LSTM-based prediction. The HELP has a higher probability to find a better hyperparameter with less time. The HELP uses a series of hyperparameters in a time period as input and predicts the fitness values of these hyperparameters. Then, exploration directions in the hyperparameter space yielding higher fitness values will have higher probabilities to be explored in the next turn. Experimental results for training both the Generative Adversarial Net and the Convolution Neural Network show that the HELP finds hyperparameters yielding better results and converges faster.
Original languageEnglish
Pages (from-to)161-172
JournalNeurocomputing
Volume442
Early online date4 Mar 2021
DOIs
Publication statusPublished - 28 Jun 2021
Externally publishedYes

Bibliographical note

This work was supported by the National Natural Science Foundation of China under Grants 61876066 and 61672443, Guangdong Province Science and Technology Plan Project (Collaborative Innovation and Platform Environment Construction) 2019A050510006, the National Training Program of Innovation and Entrepreneurship for College Students 201910561071, and Hong Kong RGC General Research Funds under Grant 9042038 (CityU 11205314) and Grant 9042322 (CityU 11200116).

Keywords

  • Convolutional neural network
  • Deep neural network
  • Generative adversarial net
  • Hyperparameter tuning

Fingerprint

Dive into the research topics of 'HELP : An LSTM-based approach to hyperparameter exploration in neural network learning'. Together they form a unique fingerprint.

Cite this