Haipeng Du

Mixed Reality (MR) telecollaboration aims to enable users to share local objects as real-time synchronized virtual replicas to remote partners and collaborate on physical tasks as if they were co-located. However, in everyday scenarios with mobile and easy-to-setup MR devices, visualizing shared objects in a single modality, ranging from 2D images to 3D reconstruction, struggles to simultaneously optimize all the aspects of Spatiality, Fidelity, and Real-time performance. To overcome this issue, existing methods explore integrating multiple visualization modalities to leverage their respective advantages in subsets of the three aspects. However, they focus on fixed modality combinations without considering user-centered task contexts and workflow, where users may prioritize different aspects of the visualization across task phases. Moreover, they lack support for switching or require manual switching across modalities, which could become disruptive and tiring. In this paper, we propose adapting object visualization based on spatio-temporal contexts in telecollaboration. Specifically, we first couple task type with the user's relative viewing distance as the spatial context, and examine its impact on users' prioritized visualization aspects, and the corresponding switching thresholds. With differing generation speeds of modalities, we then explore temporal switching schemes when the preferred modality is not immediately available. With the obtained design choices, we implement CollabVisAdapt, a proof-of-concept prototype that supports automatic adaptation of object visualization based on spatio-temporal contexts in MR telecollaboration. A user study in remote maintenance verifies the effectiveness of the proposed workflow with adaptive visualization and the usability of the system.

Optimizing quality of experience (QoE) for live video streaming (LVS) remains a long-standing challenge for content delivery network (CDN) providers. Today’s CDNs predominantly employ static congestion control (CC) configurations, yet the heterogeneity of LVS scenarios undermines the efficacy of a uniform CC solution applicable to all users, as evidenced by our production measurements. While learning-based CC approaches show promise, they often suffer from limited generalizability, non-transparent design, and high computational overhead, struggling in large-scale CDNs. In this paper, we propose BIFROST, a new CC architecture grounded in a concerndecoupling paradigm. BIFROST separates fixed control logic from scenario-specific parameter optimization, enabling CDN systems to adapt dynamically to diverse LVS scenarios. To materialize it at CDN scale, BIFROST introduces a bilateral collaboration mechanism that identifies the LVS scenarios of each session by extracting client-side characteristics from viewing requests. It further employs offline model-based optimization to derive more effective control parameters for each scenario. We have deployed BIFROST on Alibaba Cloud’s production CDN for nearly a year, serving a commercial LVS application. BIFROST has markedly improved QoE metrics by 2.7% to 32.1%, reinforcing the competitive edge in the CDN market. We also share our experiences and lessons learned from its large-scale deployment.

Continual learning for open-vocabulary object detection aims to enable pretrained vision–language detectors to adapt to diverse specialized domains while preserving their zero-shot generalization capabilities. However, existing methods primarily focus on mitigating catastrophic forgetting, often neglecting the substantial domain shifts commonly encountered in real-world applications. To address this critical oversight, we pioneer Open-Domain Continual Object Detection (OD-COD), a new paradigm that requires detectors to continually adapt across domains with significant stylistic gaps. We propose Disentangled Domain Knowledge-Aided Learning (D-Know) to tackle this challenge. This framework explicitly disentangles domain-general priors from category-specific adaptation, managing them dynamically in a scalable domain knowledge base. Specifically, D-Know first learns domain priors in a self-supervised manner and then leverages these priors to facilitate category-specific adaptation within each domain. To rigorously evaluate this task, we construct OD-CODB, the first dedicated benchmark spanning six domains with substantial visual variations. Extensive experiments demonstrate that D-Know achieves superior performance, surpassing current state-of-the-art methods by an average of 4.2% mAP under open-domain continual settings while maintaining strong zero-shot generalization. Furthermore, experiments under the few-shot setting confirm D-Know’s superior data efficiency.

Convolutional Neural Networks (CNNs) have achieved remarkable performance in computer vision tasks, but their deployment on resource-constrained devices remains challenging. While existing lightweight CNNs reduce FLOPs significantly, their practical inference speed is limited by memory access bottlenecks. We hence propose CFNet, an efficient architecture that bridges the gap between theoretical efficiency and practical speed through synergistic design of channel-focused convolution (CFConv) and channel mixed unit (CMU). CFConv dynamically selects informative channels via learnable GroupNorm scaling factors and reparameterization, reducing both FLOPs and memory access, while CMU enables cross-channel communication through a split-transform-and-mix strategy to mitigate information loss. Experiments on CIFAR/ImageNet classification and MS COCO object detection demonstrate CFNet’s superior performance. On ImageNet-1K, CFNet-A achieves 35.5% and 189.4% GPU throughput improvements over MobileNetV2 and MobileViTv1-XXS respectively, while delivering 1.76% and 4.09% accuracy gains. CFNet-E attains 83.5% top-1 accuracy, outperforming Swin-S by 0.47% with 44.6% higher GPU throughput and 43.6% lower CPU inference latency.

The proliferation of video data and advancements in Deep Neural Networks (DNNs) have greatly boosted live video analytics, driven by the growing video capture capabilities of mobile devices. However, resource limitations necessitate the transmission of endpoint-collected videos to servers for inference. To meet real-time requirements and ensure accurate inference, it is essential to adjust video configurations at the endpoint. Traditional methods rely on deterministic strategies, posing difficulties in adapting to dynamic networks and video content. Meanwhile, emerging learning-based schemes suffer from trial-and-error exploration mechanisms, resulting in a concerning long-tail effect on upload latency. In this paper, we propose a novel lightweight and robust configuration adaptation policy (LCA), which fuses heuristic and RL-based agents using multi-teacher knowledge distillation (MKD) theory. Firstly, we propose a content-sensitive and bandwidth-adaptive RL agent and introduce a Lyapunov-based optimization agent for ensuring latency robustness. To leverage both agents' strengths, we design a feature-guided multi-teacher distillation network to transfer their advantages to the student. The experimental results across two vision tasks (pose estimation and semantic segmentation) demonstrate that LCA significantly reduces transmission latency compared to prior work (average reduction of 47.11%-89.55%, 95-percentile reduction of 27.63%-88.78%) and computational overhead while maintaining comparable inference accuracy.

With the widespread deployment of edge computing, the focus has shifted to machine-centric live video streaming, where endpoint-collected videos are transmitted over networks to edge servers for analysis. Unlike maximizing user's Quality of Experience (QoE), machine-centric video streaming optimizes the machine's Quality of Inference (QoI) by balancing the inference accuracy, inference delay, and transmission latency with video adaptive configuration. Traditional heuristic configuration adaption methods are reliable but unable to respond to erratic network fluctuations. Reinforcement learning (RL) based algorithms exhibit superior flexibility but suffer from exploration mechanisms, resulting in long-tail effects on upload latency. In this paper, we propose FHVAC, which dynamically selects video encoding parameters for live streaming by coherently fusing rule-based and RL-based agent at the feature level. We initially develop a robust rule-based approach for ensuring the low latency in transmission, and employ imitation learning to convert it into a neural network equivalently. Subsequently, we design a novel module to combine the two approaches and assess various fusion mechanisms. Our evaluation of FHVAC across two vision tasks (pose estimation and semantic segmentation) in two scenarios (trace-driven simulation and testbed-based experiment) shows that FHVAC enhances the average QoI, and reduces 10.61%-65.27% latency tail performance compared to prior work.