Kaiwen Dong

Few-shot Action Recognition (FSAR) aims to recognize novel actions from only a few labeled examples, posing challenges due to limited supervision and complex temporal dynamics. Existing methods often adopt a unified motion modeling strategy for both short- and long-term dynamics, overlooking the need to adapt motion pattern extraction to the specific temporal properties inherent to different timescales. This forces models to hedge against multi-scale relevance through exhaustive searches over temporal tuples, followed by heavy spatio-temporal fusion, which substantially increases parameters and computation and ultimately limits efficiency. To this end, we propose the efficient Temporal Consistency and Variation-Guided Spatio-Temporal Aggregation Network (TCV-STA), which comprises four key components: the Temporal Consistency Module (TCM), the Temporal Variation Module (TVM), the Spatio-Temporal Aggregation attention (STA), and the Shifted Window Temporal Attention (SWTA). The TCM captures stable motion patterns to suppress short-term perturbations and enhance temporal consistency for robust motion representation, while the TVM models dynamic motion patterns to highlight long-term variations that improve inter-class discriminability and facilitate intra-class alignment. Built upon these complementary motion cues, the STA selectively aggregates spatial and temporal representations under the guidance of the learned stable and dynamic motion patterns, avoiding global dense fusion. Finally, to address the limited receptive field and discontinuous modeling caused by frame grouping in TCM and TVM, we adapt a SWTA to capture longer-range temporal dependencies and ensure smooth transitions across subaction segments for few-shot action recognition. Experiments demonstrate that TCV-STA achieves competitive accuracy across four widely-used FSAR benchmarks while reducing parameters by up to 27.9% and computational cost by 21.3%, striking a favorable balance between accuracy and efficiency for deployment in resource-constrained scenarios.

Multi-view 3-D Human Pose Estimation (HPE) effectively addresses challenges such as depth ambiguity and occlusion faced by monocular methods through the complementing of geometric information from multiple views. However, existing multiview methods often necessitate well-calibrated camera parameters or rely on complex parametric models. These requirements can result in inaccuracies when camera placement is perturbed and can negatively impact the deployability. This paper proposes a lightweight approach that synergistically models geometric information with spatial-temporal information without relying on camera parameters, named STG-GCN. We leverage the inherent connections in multiview sequences of 2-D poses, representing them as a Spatial-Temporal-Geometric graph (STG-Graph), which allows for the simultaneous encoding of spatial-temporal-geometric relations across various joints, consecutive frames, and multiple views. Using a unified graph to model all features, this approach reduces the parameter explosion in existing methods, caused by separate modules extracting spatial, temporal, and view axis features. Building upon the STG-Graph, an Adaptive Confidence-Aware Graph Convolution (ACA-GraphConv) is proposed to mitigate the impact of unreliable 2-D poses predicted by 2-D pose estimators. This is achieved by leveraging corresponding confidence scores to adjust the graph convolution accordingly. Experimental results on two public datasets demonstrate that our STG-GCN achieves performance comparable to state-of-the-art approaches while significantly reducing parameter volume. Ablation studies also illustrate the effectiveness of our ACA-GraphConv in both monocular and multiview scenarios.

Edge video analytics enables agile responses of machine-centric applications by streaming videos from end devices to edge servers (ESs) for resource-intensive Deep Neural Network (DNN) inference. Quality of Inference (QoI), reflected by end-to-end analytics delay and inference accuracy, is crucial for the high-quality real-time decision-making. Due to the limited ES resources and dynamic network conditions, video configuration adaptation and fine-grained resource allocation are necessary to enhance the QoI of video streams, which involves a delicate balance between delay and accuracy. In this paper, we propose an edge-enabled multi-video analytics framework, in which performing DNN inference relies on heterogeneous resources, including CPU, GPU, and memory. Considering the different impacts of heterogeneous resources on QoI and double-queue backlogs, we formulate the joint video configuration adaptation, CPU-GPU resource allocation, and batch size selection (JCCGB) problem based on semi-Markov Decision Process to maximize the long-term average QoI. Then, Lyapunov optimization is applied to transform the original problem into one that minimizes the Lyapunov drift-plus-penalty upper bound, thereby ensuring the stability of both the transmission and computation queues. To tackle the potential multimodality of the optimal configuration adaptation and resource allocation policy, we propose the diffusion deep reinforcement learning based JCCGB (DD-JCCGB) scheme. This approach effectively enhances the policy performance and accelerates the training process. Experiments driven by real-world network traces demonstrate that the DD-JCCGB algorithm improves the long-term average QoI and outperforms baseline schemes.

Unsupervised 3D pose estimation has gained prominence due to the challenges in acquiring labeled 3D data for training. Despite promising progress, unsupervised approaches still lag behind supervised methods in performance. Two factors impede the progress of unsupervised approaches: incomplete geometric constraint and inadequate interaction among spatial, temporal, and multi-view features. This paper introduces an unsupervised pipeline that uses calibrated camera parameters as geometric constraints across views and coordinate spaces to optimize the model by minimizing inconsistencies between the 2D input pose and the re-projection of the predicted 3D pose. This pipeline utilizes the novel Hierarchical Cross Transformer (HCT) to encode higher levels of information by enabling interactions among hierarchical features containing different level of temporal, spatial and cross-view information. By minimizing the reliance on human-specific parts, the HCT shows potential for adapting to various pose estimation tasks. To validate the adaptability, we build a connection between human pose estimation and scene pose estimation, introducing Dynamic-Keypoints-3D (DK-3D) dataset tailored for 3D Scene Pose Estimation in robotic manipulation. Experiments on two 3D human pose estimation datasets demonstrate our method’s new state-of-the-art performance among weakly and unsupervised approaches. The adaptability of our method is confirmed through experiments on DK-3D, setting the initial benchmark for unsupervised 2D-to-3D scene pose lifting.

Traditional passive point cloud acquisition systems, such as lidars or stereo cameras, can be impractical in real-life and industrial use cases. Firstly, some extreme environments may preclude the use of these sensors. Secondly, they capture information from the entire scene instead of focusing on areas relevant to the end task, such as object recognition and localization. In contrast, we propose to train a Reinforcement Learning (RL) agent with dual objectives: i) control a robot equipped with a tactile (or laser) sensor to iteratively collect a few relevant points from the scene, and ii) recognize and localize objects from the sparse point cloud which has been collected. The iterative point sampling strategy, referred to as an active sampling strategy, is jointly trained with the classifier and the pose estimator to ensure efficient exploration that focuses on areas relevant to the recognition task. To achive these two objectives, we introduce three RL reward terms: classification, exploration, and pose estimation rewards. These rewards serve the purpose of offering guidance and supervision in their respective domain, allowing us to delve into their individual impacts and contributions. We compare the proposed framework to both active sampling strategies and passive hard-coded sampling strategies coupled with stateof- the-art point cloud classifiers. Furthermore, we evaluate our framework in realistic scenarios, considering realistic and similar objects, as well as accounting for uncertainty in the object's position in the workspace.