RPRenjie Pan

Renjie Pan

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Renjie Pan has a total of 15 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Match Any Keypoints

    by Haobo Xu, Renjie Pan, Jun Zhou, Hua Yang, Cunyan Li

    Previous research on sparse feature matching typically involves a staged optimization process of keypoint detection, description, and matching. While it allows the network to adapt to specific inputs, it may limit the network's expressive capability and the overall architectural flexibility. In this study, we rethink the matching framework and propose to directly match any given keypoints, optimizing the matching network in an approximately end-to-end manner. To achieve this, firstly, we dynamically sample random positions within the images as assumed keypoints during training, allowing the network to explore a broader matching space. Secondly, we replace specific descriptors with high-efficiency sparse embeddings at multi levels of the image, facilitating the direct learning of underlying textures. Thirdly, we propose a novel and promising architecture, called Proposal-Guided TRansformer (PGTR), which aggregates context information from neighboring match proposals instead of searching globally with local features. PGTR works especially well under our training approach, and attain a synergistic advantage in terms of performance and efficiency. The overall pipeline achieves outstanding performance on various keypoints without any retraining, and can be flexibly reused when new keypoints emerge, making it valuable for real-world applications. Code will be available.

  • Psychology-informed safety attributes recognition in dense crowds

    by Jiaqi Yu, Yanshan Zhou, Renjie Pan, Cunyan Li, Hua Yang


  • Hydrodynamics Regularization in Reinforcement Learning for Navigating Crowded Scenarios

    by Pingrui Lai, Renjie Pan, Jiaqi Yu, Hua Yang

    The navigation task in dense crowds is a key research problem in real-world scenarios. It requires an agent to avoid collisions in dynamic environments and reach the agent's destination, ensuring high accuracy and efficiency in its decisions. Existing methods typically treat pedestrians as rigid bodies, detect object bounding boxes, and use rigid body dynamics to guide agent behavior. However, in densely crowded scenarios, this approach may lead to suboptimal path planning solutions, thereby imposing more stringent constraints on the agent's action space. In some real-world navigation scenarios, pedestrians can avoid collisions with minor posture adjustments without having to change their direction. In this letter, we propose Hydrodynamics Regularization to address the challenges posed by the modeling of pedestrians in dense crowd environments. This method treats pedestrians as deformable bodies and leverages fluid dynamics equations to compute socially compliant velocities and postures for the agent in dense crowds. By introducing soft constraints on the action space, it effectively reduces collisions between the agent and surrounding obstacles. We validate the effectiveness of Hydrodynamics Regularization in reinforcement learning navigation problems with partially observable Markov processes. Extensive experiments demonstrate that Hydrodynamics Regularization effectively mitigates collisions caused by the modeling approach, especially in dense crowd environments, allowing the navigation agent to achieve higher success rates, fewer collisions, and shorter completion times. The Hydrodynamics Regularization module is a plug-and-play component that can be seamlessly integrated into any reinforcement learning algorithm, demonstrating excellent generalizability.


  • Open Access

    ReAL : Improving Image-Text Retrieval with Authentic Negative Repository Learning

    by Renjie Pan, Hua Yang, Xiangyu Zhao

    Current methods for image-text retrieval commonly propose various fusion modules to achieve robust visual-textual alignment, primarily relying on in-batch learning to guide the matching process. Some follow-up methods seek to enlarge the number of negative samples to boost image-text contrastive learning. However, these methods often face challenges posed by semantic-consistent negatives, i.e., negatives samples that share correspondence with the ground truth, leading to confusion in learning cross-modal semantics. To address this issue, we propose a novel Re trieve with A uthentic negative repository L earning ( ReAL ) method, which constructs a specific Authentic Negative Repository filled with valuable negative sample pairs. By introducing a Unique Negative Filter with a Discriminative Triplet Ranking Loss, ReAL effectively filters out the semantic-consistent negatives through similarity distribution analysis and threshold learning. Moreover, existing fusion paradigms suffer from intricate use of fine-grained representations from word- and region-level instances to progressively refine the fused embedding. In this paper, we propose a lightweight Cluster Refinement Module to exploit cross-modal semantics in a 1-way-1-out paradigm. Each visual-textual alignment can spontaneously uncover correlations with adjacent alignments through aggregation and re-allocation, without the need for a redundant and cost-inefficient refinement stage. Furthermore, ReAL employs dual momentum encoders with two memory banks, expanding the selection range of the Authentic Negative Repository to include a broader set of negatives. Extensive experiments conducted on Flickr30K, MS-COCO, and the augmented Flickr30K (with more hard negatives) demonstrate the superiority and robustness of ReAL, while also showcasing its significantly reduced inference time compared to other competitive baselines.


  • Open Access

    Psychology-Guided Environment Aware Network for Discovering Social Interaction Groups from Videos

    by Jiaqi Yu, Jinhai Yang, Hua Yang, Renjie Pan, Pingrui Lai, Guangtao Zhai

    Social interaction is a common phenomenon in human societies. Different from discovering groups based on the similarity of individuals’ actions, social interaction focuses more on the mutual influence between people. Although people can easily judge whether or not there are social interactions in a real-world scene, it is difficult for an intelligent system to discover social interactions. Initiating and concluding social interactions are greatly influenced by an individual’s social cognition and the surrounding environment, which are closely related to psychology. Thus, converting the psychological factors that impact social interactions into quantifiable visual representations and creating a model for interaction relationships poses a significant challenge. To this end, we propose a Psychology-Guided Environment Aware Network (PEAN) that models social interaction among people in videos using supervised learning. Specifically, we divide the surrounding environment into scene-aware visual-based and human-aware visual-based descriptions. For the scene-aware visual clue, we utilize 3D features as global visual representations. For the human-aware visual clue, we consider instance-based location and behaviour-related visual representations to map human-centred interaction elements in social psychology: distance, openness, and orientation. In addition, we design an environment aware mechanism to integrate features from visual clues, with a Transformer to explore the relation between individuals and construct pairwise interaction strength features. The interaction intensity matrix reflecting the mutual nature of the interaction is obtained by processing the interaction strength features with the interaction discovery module. An interaction constrained loss function composed of interaction critical loss function and smoothFβloss function is proposed to optimize the whole framework to improve the distinction of the interaction matrix and alleviate class imbalance caused by pairwise interaction sparsity. Given the diversity of real-world interactions, we collect a new dataset named Social Basketball Activity Dataset (Soical-BAD), covering complex social interactions. Our method achieves the best performance among social-CAD, social-BAD, and their combined dataset named Video Social Interaction Dataset (VSID).


  • Joint Intra & Inter-Grained Reasoning: A New Look Into Semantic Consistency of Image-Text Retrieval

    by Renjie Pan, Hua Yang, Cunyan Li, Jinhai Yang

    Multimodal understanding aims at constructing semantic correlations among modalities of data while performing various downstream tasks. As one of the primary multimodal downstream tasks, image-text retrieval imposes a high demand on semantic alignment because of the independent expression paradigms of images and text. Existing methods mainly construct a joint embedding space at a single granularity level (either global or local). However, such single reasoning paradigms lack granularity interaction, resulting in semantic inconsistency and cross-domain catastrophes. To address these issues, we design a novel Joint Intra and Inter-grained Network (JIIGNet), focusing on not only intra- but also inter-grained interaction between modalities by combining scene information (global) with region-level (local) instances. Specifically, we simultaneously initiate three specific alignment modules, i.e., global-grained, local-grained, and cross-grained alignment modules, followed by Triplet Attention Refinement to better refine the fused embedding at the alignment-level with proper self and cross attention. For different scenarios, a Style Adaptation Head is further designed to smartly accommodate different samples. We validate JIIGNet through extensive experiments conducted on two widely used datasets: Flickr-30 K and MS-COCO, demonstrating the effectiveness of our proposed method.


  • Attentive Feature Focusing for Person Search by Natural Language

    by Renjie Pan, Hua Yang, Xinxin Yang, Mao Xiaodong, Long Ye, Da Pan