Skip to main navigation Skip to search Skip to main content

Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Framework

  • Yinxi TIAN
  • , Changwu HUANG*
  • , Ke TANG*
  • , Xin YAO
  • *Corresponding author for this work

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Referred Conference Paper › peer-review

Abstract

Knowledge distillation (KD) transfers knowledge from large teacher models to compact student models, enabling efficient deployment on resource-constrained devices. While diverse KD methods–including response-based, feature-based, and relation-based approaches–capture different aspects of teacher knowledge, integrating multiple methods or knowledge sources is promising but often hampered by complex implementation, inflexible combinations, and catastrophic forgetting, limiting practical effectiveness. This work proposes SMSKD (Sequential Multi-Stage Knowledge Distillation), a flexible framework that sequentially integrates heterogeneous KD methods. At each stage, the student is trained with a specific distillation method, while a frozen reference model from the previous stage anchors learned knowledge to mitigate forgetting. Additionally, we introduce an adaptive weighting mechanism based on the teacher’s true class probability (TCP) that dynamically adjusts the reference loss per sample to balance knowledge retention and integration. By design, SMSKD supports arbitrary method combinations and stage counts with negligible computational overhead. Extensive experiments show that SMSKD consistently improves student accuracy across diverse teacher–student architectures and method combinations, outperforming existing baselines. Ablation studies confirm that stage-wise distillation and reference model supervision are primary contributors to performance gains, with TCP-based adaptive weighting providing complementary benefits. Overall, SMSKD is a practical and resource-efficient solution for integrating heterogeneous KD methods.

Original languageEnglish
Title of host publicationProceedings of 18th International Conference on Machine Learning and Computing, ICMLC2026
EditorsLin HUANG, David GREENHALGH
PublisherSpringer Science and Business Media Deutschland GmbH
Pages991-1009
Number of pages19
Volume2
ISBN (Electronic)9783032312365
ISBN (Print)9783032312358
DOIs
Publication statusE-pub ahead of print - 2 Sept 2026
Event18th International Conference on Machine Learning and Computing, ICMLC2026 - Nanjing, China
Duration: 6 Feb 2026 → 9 Feb 2026

Publication series

NameLecture Notes in Networks and Systems
PublisherSpringer
Volume2061
ISSN (Print)2367-3370
ISSN (Electronic)2367-3389

Conference

Conference18th International Conference on Machine Learning and Computing, ICMLC2026
Country/TerritoryChina
CityNanjing
Period6/02/26 → 9/02/26

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2027.

Funding

This work was supported by the National Natural Science Foundation of China (Grant No. 62250710682), an internal grant of Lingnan University, the Guangdong Provincial Key Laboratory (Grant No. 2020B121201001), and the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (Grant No. 2017ZT07X386).

Keywords

  • Deep Neural Networks
  • Knowledge Distillation
  • Knowledge Integration

Fingerprint

Dive into the research topics of 'Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Framework'. Together they form a unique fingerprint.

Cite this