Abstract
Semantic segmentation is a critical perception task for real-time scene understanding in consumer-grade edge devices, particularly in green transportation systems such as autonomous vehicles, smart dashcams, and in-vehicle ADAS units. However, the high computational cost of state-of-the-art models hinders their deployment in resource-constrained scenarios. Knowledge distillation offers a solution by transferring knowledge from a large teacher model to a compact student. Yet the prevalent multi-head architecture in modern teachers, which includes a main head together with auxiliary heads, points to a richer knowledge source than any single prediction. The interactions among these heads may encode valuable structural cues about class relationships and contextual dependencies, which are often overlooked in conventional distillation methods. Effectively distilling this cross-head organizational information, especially under architectural asymmetry, remains a significant challenge. To address this, we propose Cross-Head Relation Distillation (CHRD), a framework that explicitly models and transfers the structured knowledge derived from the teacher’s dual-head system. CHRD introduces cross-head consistency imitation to transfer a confidence-weighted consensus and cross-head relation imitation to capture class-level affinities. To address the structural asymmetry between teacher and student, we design a shared auxiliary head mechanism, enabling any student to leverage this cross-head knowledge. Furthermore, a loss normalization strategy ensures stable, progressive learning. Our method achieves competitive performance across multiple benchmarks, setting a new state-of-the-art on ADE20K and attaining highly competitive results on Cityscapes and CamVid, without relying on intermediate feature alignment. Code will be available at https://github.com/Fateeeeee/CHRD.
Funding Information
  • National Natural Science Foundation of China (62576269)

This publication has 46 references indexed in Scilit: