A new initialization method based on normed statistical spaces in deep networks

Hongfei YANG, Xiaofeng DING, Raymond CHAN, Hui HU, Yaxin PENG, Tieyong ZENG*

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

6 Citations (Scopus)

Abstract

Training deep neural networks can be difficult. For classical neural networks, the initialization method by Xavier and Yoshua which is later generalized by He, Zhang, Ren and Sun can facilitate stable training. How-ever, with the recent development of new layer types, we find that the above mentioned initialization methods may fail to lead to successful training. Based on these two methods, we will propose a new initialization by studying the parameter space of a network. Our principal is to put constrains on the growth of parameters in different layers in a consistent way. In order to do so, we introduce a norm to the parameter space and use this norm to measure the growth of parameters. Our new method is suitable for a wide range of layer types, especially for layers with parameter-sharing weight matrices.

Original languageEnglish
Pages (from-to)147-158
Number of pages12
JournalInverse Problems and Imaging
Volume15
Issue number1
DOIs
Publication statusPublished - Feb 2021
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2021, American Institute of Mathematical Sciences. All rights reserved.

Funding

2020 Mathematics Subject Classification. Primary: 68T01, 68T05; Secondary: 68Q32. Key words and phrases. Neural networks, parameters sharing, parameters initialization, deep learning, model training. Raymond Chan’s research is supported by HKRGC Grants No. CUHK 14306316 and CUHK 14301718, CityU Grant 9380101, CRF Grant C1007-15G, AoE/M-05/12. Tieyong Zeng’s research is supported by National Science Foundation of China No. 11671002, CUHK start-up and CUHK DAG 4053342, RGC 14300219, and NSFC/RGC N CUHK 415/19.

Keywords

  • Deep learning
  • Model training
  • Neural networks
  • Parameters initialization
  • Parameters sharing

Fingerprint

Dive into the research topics of 'A new initialization method based on normed statistical spaces in deep networks'. Together they form a unique fingerprint.

Cite this