A new initialization method based on normed statistical spaces in deep networks

Hongfei YANG, Xiaofeng DING, Raymond CHAN, Hui HU, Yaxin PENG, Tieyong ZENG*

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

6 Citations (Scopus)

Abstract

Training deep neural networks can be difficult. For classical neural networks, the initialization method by Xavier and Yoshua which is later generalized by He, Zhang, Ren and Sun can facilitate stable training. How-ever, with the recent development of new layer types, we find that the above mentioned initialization methods may fail to lead to successful training. Based on these two methods, we will propose a new initialization by studying the parameter space of a network. Our principal is to put constrains on the growth of parameters in different layers in a consistent way. In order to do so, we introduce a norm to the parameter space and use this norm to measure the growth of parameters. Our new method is suitable for a wide range of layer types, especially for layers with parameter-sharing weight matrices.

Original languageEnglish
Pages (from-to)147-158
Number of pages12
JournalInverse Problems and Imaging
Volume15
Issue number1
DOIs
Publication statusPublished - Feb 2021
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2021, American Institute of Mathematical Sciences. All rights reserved.

Keywords

  • Deep learning
  • Model training
  • Neural networks
  • Parameters initialization
  • Parameters sharing

Fingerprint

Dive into the research topics of 'A new initialization method based on normed statistical spaces in deep networks'. Together they form a unique fingerprint.

Cite this