A New Initialization Method for Neural Networks with Weight Sharing

Xiaofeng DING; Hongfei YANG; Hui HU; Raymond H. CHAN; Yaxin PENG; Tieyong ZENG

doi:10.1007/978-981-16-2701-9_9

A New Initialization Method for Neural Networks with Weight Sharing

Xiaofeng DING, Hongfei YANG, Hui HU, Raymond H. CHAN, Yaxin PENG, Tieyong ZENG^*

^*Corresponding author for this work

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Research › peer-review

1 Citation (Scopus)

Abstract

A proper initialization of parameters in a neural network can facilitate its training. The Xavier initialization introduced by Glorot and Bengio which is later generalized to Kaiming initialization by He, Zhang, Ren and Sun are now widely used. However, from experiments we find that networks with heavy weight sharing are difficulty to train even with the Xavier or the Kaiming initialization. We also notice that a certain simple network can be decomposed in two ways, where one is difficult to train while the other is easy to train, when both are properly initialized by the Xavier or the Kaiming initialization. In this paper we will propose a new initialization method which will increase training speed and training stability of neural networks with heavy weight sharing. We will also propose a simple yet efficient method to adjust learning rates layer by layer which is indispensable to our initialization.

Original language	English
Title of host publication	Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018
Editors	Xue-Cheng TAI, Suhua WEI, Haiguang LIU
Publisher	Springer
Pages	165-179
Number of pages	15
ISBN (Electronic)	9789811627019
ISBN (Print)	9789811627002
DOIs	https://doi.org/10.1007/978-981-16-2701-9_9
Publication status	Published - 2021
Externally published	Yes
Event	International Workshop on Image Processing and Inverse Problems, IPIP 2018 - Beijing, China Duration: 21 Apr 2018 → 24 Apr 2018

Publication series

Name	Springer Proceedings in Mathematics and Statistics
Volume	360
ISSN (Print)	2194-1009
ISSN (Electronic)	2194-1017

Conference

Conference	International Workshop on Image Processing and Inverse Problems, IPIP 2018
Country/Territory	China
City	Beijing
Period	21/04/18 → 24/04/18

Bibliographical note

Publisher Copyright:
© 2021, Springer Nature Singapore Pte Ltd.

Keywords

Learning rate
Neural networks
Weight sharing
Xavier initialization

Access to Document

10.1007/978-981-16-2701-9_9

Cite this

DING, X., YANG, H., HU, H., CHAN, R. H., PENG, Y., & ZENG, T. (2021). A New Initialization Method for Neural Networks with Weight Sharing. In X.-C. TAI, S. WEI, & H. LIU (Eds.), Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018 (pp. 165-179). (Springer Proceedings in Mathematics and Statistics; Vol. 360). Springer. https://doi.org/10.1007/978-981-16-2701-9_9

@inproceedings{826c0a4de79746ff8a5d798940e0c4ac,

title = "A New Initialization Method for Neural Networks with Weight Sharing",

abstract = "A proper initialization of parameters in a neural network can facilitate its training. The Xavier initialization introduced by Glorot and Bengio which is later generalized to Kaiming initialization by He, Zhang, Ren and Sun are now widely used. However, from experiments we find that networks with heavy weight sharing are difficulty to train even with the Xavier or the Kaiming initialization. We also notice that a certain simple network can be decomposed in two ways, where one is difficult to train while the other is easy to train, when both are properly initialized by the Xavier or the Kaiming initialization. In this paper we will propose a new initialization method which will increase training speed and training stability of neural networks with heavy weight sharing. We will also propose a simple yet efficient method to adjust learning rates layer by layer which is indispensable to our initialization.",

keywords = "Learning rate, Neural networks, Weight sharing, Xavier initialization",

author = "Xiaofeng DING and Hongfei YANG and Hui HU and CHAN, {Raymond H.} and Yaxin PENG and Tieyong ZENG",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Singapore Pte Ltd.; International Workshop on Image Processing and Inverse Problems, IPIP 2018 ; Conference date: 21-04-2018 Through 24-04-2018",

year = "2021",

doi = "10.1007/978-981-16-2701-9_9",

language = "English",

isbn = "9789811627002",

series = "Springer Proceedings in Mathematics and Statistics",

publisher = "Springer",

pages = "165--179",

editor = "Xue-Cheng TAI and Suhua WEI and Haiguang LIU",

booktitle = "Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018",

address = "Germany",

}

DING, X, YANG, H, HU, H, CHAN, RH, PENG, Y & ZENG, T 2021, A New Initialization Method for Neural Networks with Weight Sharing. in X-C TAI, S WEI & H LIU (eds), Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018. Springer Proceedings in Mathematics and Statistics, vol. 360, Springer, pp. 165-179, International Workshop on Image Processing and Inverse Problems, IPIP 2018, Beijing, China, 21/04/18. https://doi.org/10.1007/978-981-16-2701-9_9

A New Initialization Method for Neural Networks with Weight Sharing. / DING, Xiaofeng; YANG, Hongfei; HU, Hui et al.
Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018. ed. / Xue-Cheng TAI; Suhua WEI; Haiguang LIU. Springer, 2021. p. 165-179 (Springer Proceedings in Mathematics and Statistics; Vol. 360).

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Research › peer-review

TY - GEN

T1 - A New Initialization Method for Neural Networks with Weight Sharing

AU - DING, Xiaofeng

AU - YANG, Hongfei

AU - HU, Hui

AU - CHAN, Raymond H.

AU - PENG, Yaxin

AU - ZENG, Tieyong

PY - 2021

Y1 - 2021

N2 - A proper initialization of parameters in a neural network can facilitate its training. The Xavier initialization introduced by Glorot and Bengio which is later generalized to Kaiming initialization by He, Zhang, Ren and Sun are now widely used. However, from experiments we find that networks with heavy weight sharing are difficulty to train even with the Xavier or the Kaiming initialization. We also notice that a certain simple network can be decomposed in two ways, where one is difficult to train while the other is easy to train, when both are properly initialized by the Xavier or the Kaiming initialization. In this paper we will propose a new initialization method which will increase training speed and training stability of neural networks with heavy weight sharing. We will also propose a simple yet efficient method to adjust learning rates layer by layer which is indispensable to our initialization.

AB - A proper initialization of parameters in a neural network can facilitate its training. The Xavier initialization introduced by Glorot and Bengio which is later generalized to Kaiming initialization by He, Zhang, Ren and Sun are now widely used. However, from experiments we find that networks with heavy weight sharing are difficulty to train even with the Xavier or the Kaiming initialization. We also notice that a certain simple network can be decomposed in two ways, where one is difficult to train while the other is easy to train, when both are properly initialized by the Xavier or the Kaiming initialization. In this paper we will propose a new initialization method which will increase training speed and training stability of neural networks with heavy weight sharing. We will also propose a simple yet efficient method to adjust learning rates layer by layer which is indispensable to our initialization.

KW - Learning rate

KW - Neural networks

KW - Weight sharing

KW - Xavier initialization

UR - http://www.scopus.com/inward/record.url?scp=85116465564&partnerID=8YFLogxK

U2 - 10.1007/978-981-16-2701-9_9

DO - 10.1007/978-981-16-2701-9_9

M3 - Conference paper (refereed)

AN - SCOPUS:85116465564

SN - 9789811627002

T3 - Springer Proceedings in Mathematics and Statistics

SP - 165

EP - 179

BT - Mathematical Methods in Image Processing and Inverse Problems, IPIP 2018

A2 - TAI, Xue-Cheng

A2 - WEI, Suhua

A2 - LIU, Haiguang

PB - Springer

T2 - International Workshop on Image Processing and Inverse Problems, IPIP 2018

Y2 - 21 April 2018 through 24 April 2018

ER -

A New Initialization Method for Neural Networks with Weight Sharing

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this