Abstract
Training deep neural networks can be difficult. For classical neural networks, the initialization method by Xavier and Yoshua which is later generalized by He, Zhang, Ren and Sun can facilitate stable training. How-ever, with the recent development of new layer types, we find that the above mentioned initialization methods may fail to lead to successful training. Based on these two methods, we will propose a new initialization by studying the parameter space of a network. Our principal is to put constrains on the growth of parameters in different layers in a consistent way. In order to do so, we introduce a norm to the parameter space and use this norm to measure the growth of parameters. Our new method is suitable for a wide range of layer types, especially for layers with parameter-sharing weight matrices.
Original language | English |
---|---|
Pages (from-to) | 147-158 |
Number of pages | 12 |
Journal | Inverse Problems and Imaging |
Volume | 15 |
Issue number | 1 |
DOIs | |
Publication status | Published - Feb 2021 |
Externally published | Yes |
Bibliographical note
Publisher Copyright:© 2021, American Institute of Mathematical Sciences. All rights reserved.
Funding
2020 Mathematics Subject Classification. Primary: 68T01, 68T05; Secondary: 68Q32. Key words and phrases. Neural networks, parameters sharing, parameters initialization, deep learning, model training. Raymond Chan’s research is supported by HKRGC Grants No. CUHK 14306316 and CUHK 14301718, CityU Grant 9380101, CRF Grant C1007-15G, AoE/M-05/12. Tieyong Zeng’s research is supported by National Science Foundation of China No. 11671002, CUHK start-up and CUHK DAG 4053342, RGC 14300219, and NSFC/RGC N CUHK 415/19.
Keywords
- Deep learning
- Model training
- Neural networks
- Parameters initialization
- Parameters sharing