Efficient Robustness for Small Models via Dual Adversarial Distillation with Hybrid Supervision

Small models, despite their computational efficiency for real-time and edge applications, remain vulnerable to adversarial attacks. Adversarial Distillation (AD) has proven effective in enhancing model robustness, yet current approaches predominantly rely on single-strategy adversarial samples with either fixed or adaptive teacher supervision. Fixed supervision often leads to over-smoothing due to static guidance, while adaptive supervision incurs higher computational costs and convergence instability. To address these limitations, we propose Dual Adversarial Distillation (DAD), a novel framework that synergistically combines fixed and adaptive supervision through multi-intensity adversarial samples, with customized distillation strategies designed to enhance knowledge diversity and feature transfer. For fixed supervision, a Mixup-based strategy is employed to diversify teacher knowledge, ensuring robust feature representations by aligning the student model with the teacher's rich representations through cross-strength feature consistency. For adaptive supervision, learning diversity is augmented by integrating transformed features from multiple projection modules, effectively minimizing the feature distribution gap between teacher and student models. Computational efficiency is optimized through variable iteration strategies. Extensive experiments demonstrate the effectiveness of our method in improving the robustness of small models, achieving state-of-the-art baselines.