Cross-Head Relation Distillation for Edge-Friendly Semantic Segmentation in Transportation Systems
- 31 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 72 (2) , 4858-4871
Abstract
Semantic segmentation is a critical perception task for real-time scene understanding in consumer-grade edge devices, particularly in green transportation systems such as autonomous vehicles, smart dashcams, and in-vehicle ADAS units. However, the high computational cost of state-of-the-art models hinders their deployment in resource-constrained scenarios. Knowledge distillation offers a solution by transferring knowledge from a large teacher model to a compact student. Yet the prevalent multi-head architecture in modern teachers, which includes a main head together with auxiliary heads, points to a richer knowledge source than any single prediction. The interactions among these heads may encode valuable structural cues about class relationships and contextual dependencies, which are often overlooked in conventional distillation methods. Effectively distilling this cross-head organizational information, especially under architectural asymmetry, remains a significant challenge. To address this, we propose Cross-Head Relation Distillation (CHRD), a framework that explicitly models and transfers the structured knowledge derived from the teacher’s dual-head system. CHRD introduces cross-head consistency imitation to transfer a confidence-weighted consensus and cross-head relation imitation to capture class-level affinities. To address the structural asymmetry between teacher and student, we design a shared auxiliary head mechanism, enabling any student to leverage this cross-head knowledge. Furthermore, a loss normalization strategy ensures stable, progressive learning. Our method achieves competitive performance across multiple benchmarks, setting a new state-of-the-art on ADE20K and attaining highly competitive results on Cityscapes and CamVid, without relying on intermediate feature alignment. Code will be available at https://github.com/Fateeeeee/CHRD.Keywords
Funding Information
- National Natural Science Foundation of China (62576269)
This publication has 46 references indexed in Scilit:
- Uncertainty-Aware and Decoupled Distillation for Semantic SegmentationInternational Journal of Computer Vision, 2025
- Enhancing knowledge distillation for semantic segmentation through text-assisted modular pluginsPattern Recognition, 2025
- Region-aware mutual relational knowledge distillation for semantic segmentationPattern Recognition, 2024
- Relative difficulty distillation for semantic segmentationScience China Information Sciences, 2024
- P2AT: Pyramid pooling axial transformer for real-time semantic segmentationExpert Systems with Applications, 2024
- AutoSegEdge: Searching for the edge device real-time semantic segmentation based on multi-task learningImage and Vision Computing, 2023
- HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors2022
- BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic SegmentationInternational Journal of Computer Vision, 2021
- Intra-class Feature Variation Distillation for Semantic Segmentation2020
- ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation2018