Deep neural networks (DNNs) have shown great success in completing complex tasks. However, DNNs inevitably bring high computational cost and storage consumption due to the complexity of hierarchical structures, thereby hindering their wide deployment in Internet-of-Things (IoT) devices, which have limited computational capability and storage capacity. Therefore, it is a necessity to investigate the technologies to compact DNNs. Despite tremendous advances in compacting DNNs, few surveys summarize compacting-DNNs technologies, especially for IoT applications. Hence, this article presents a comprehensive study on compacting-DNNs technologies. We categorize compacting-DNNs technologies into three major types: 1) network model compression; 2) knowledge distillation (KD); and 3) modification of network structures. We also elaborate on the diversity of these approaches and make side-by-side comparisons. Moreover, we discuss the applications of compacted DNNs in various IoT applications and outline future directions.
Bibliographical noteFunding Information:
Manuscript received September 29, 2020; revised February 6, 2021; accepted February 28, 2021. Date of publication March 3, 2021; date of current version July 23, 2021. This work was supported in part by the Sichuan Science and Technology Program under Grant 2019YFG0405; in part by the Project of Science and Technology on Electronic Information Control Laboratory; in part by the Joint Key Research and Development Project between Sichuan and Chongqing under Grant cstc2020jscx-cylhX0004; and in part by the Macao Science and Technology Development Fund under Macao Funding Scheme for Key Research and Development Projects under Grant 0025/2019/AKP. (Corresponding author: Ke Zhang.) Ke Zhang and Yuanyuan Peng are with the School of Computer Science and Engineering, and Science and Technology on Electronic Information Control Laboratory, and also with the School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China (e-mail: firstname.lastname@example.org; email@example.com).
© 2014 IEEE.
- Deep learning (DL)
- deep neural networks (DNNs)
- Internet of Things (IoT)
- model compression