Lianyu Hu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Lianyu Hu has a total of 41 co-authors. Here are the top co-authors by number of shared publications:
- Wei Feng
School of Computer Science and Technology, Tianjin University, Tianjin, China
10 shared publications - Liqing Gao
School of Computer Science and Technology, Tiangong University, Tianjin 300387, China
5 shared publications - Zekang Liu
College of Intelligence and Computing, Tianjin University, Tianjin, China
4 shared publications - Liang Wan
Tianjin University, Tianjin, China
3 shared publications - Chi-Man Pun
University of Macau
2 shared publications - Fanhua Shang
Tianjin University, Tianjin, China
2 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 13704-13713
Reading Between the Pixels: An Inscriptive Jailbreak Attack on Text-to-Image Models
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 13704-13713
Open AccessReading Between the Pixels: An Inscriptive Jailbreak Attack on Text-to-Image Models
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 16190-16198
COMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 16190-16198
Open AccessCOMET: Contrastive Motion-Enhanced Temporal Reasoning for Video Multimodal Large Language Models
- 12 September 2026
- Published by Springer Nature
- p. 526-544
CoLT: Teaching Multi-modal Models to Think with Chain of Latent Thoughts
- 12 September 2026
- Published by Springer Nature
- p. 526-544
CoLT: Teaching Multi-modal Models to Think with Chain of Latent Thoughts
- Open Access
- 6 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-8
Gloss-Free Sign Language Translation With Optical-Flow Guided Two-Stream Network
- 6 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-8
Open AccessGloss-Free Sign Language Translation With Optical-Flow Guided Two-Stream Network
- 3 March 2025
- Published by Springer Nature in Multimedia Systems
- Vol. 31 (2) , 1-11
Rethinking the temporal downsampling paradigm for continuous sign language recognition
- 3 March 2025
- Published by Springer Nature in Multimedia Systems
- Vol. 31 (2) , 1-11
Rethinking the temporal downsampling paradigm for continuous sign language recognition
- 4 January 2025
- Published by Springer Nature in The Visual Computer
- Vol. 41 (4) , 2755-2766
DBL-SC: background-independent sign language recognition based on spatial channel separation computation
- 4 January 2025
- Published by Springer Nature in The Visual Computer
- Vol. 41 (4) , 2755-2766
DBL-SC: background-independent sign language recognition based on spatial channel separation computation
- 29 October 2024
- Published by Springer Nature
- p. 392-409
Pose-Guided Fine-Grained Sign Language Video Generation
- 29 October 2024
- Published by Springer Nature
- p. 392-409
Pose-Guided Fine-Grained Sign Language Video Generation
- Open Access
CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation
In sign language, the conveyance of human body trajectories predominantly relies upon the coordinated movements of hands and facial expressions across successive frames. Despite the recent advancements of sign language understanding methods, they often solely focus on individual frames, inevitably overlooking the inter-frame correlations that are essential for effectively modeling human body trajectories. To address this limitation, this paper introduces a spatial-temporal correlation network, denoted as CorrNet+, which explicitly identifies body trajectories across multiple frames. In specific, CorrNet+ employs a correlation module and an identification module to build human body trajectories. Afterwards, a temporal attention module is followed to adaptively evaluate the contributions of different frames. The resultant features offer a holistic perspective on human body movements, facilitating a deeper understanding of sign language. As a unified model, CorrNet+ achieves new state-of-the-art performance on two extensive sign language understanding tasks, including continuous sign language recognition (CSLR) and sign language translation (SLT). Especially, CorrNet+ surpasses previous methods equipped with resource-intensive pose-estimation networks or pre-extracted heatmaps for hand and facial feature extraction. Compared with CorrNet, CorrNet+ achieves a significant performance boost across all benchmarks while halving the computational overhead. A comprehensive comparison with previous spatial-temporal reasoning methods verifies the superiority of CorrNet+. Code is available at https://github.com/hulianyuyy/CorrNet_Plus.
Open AccessCorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation
In sign language, the conveyance of human body trajectories predominantly relies upon the coordinated movements of hands and facial expressions across successive frames. Despite the recent advancements of sign language understanding methods, they often solely focus on individual frames, inevitably overlooking the inter-frame correlations that are essential for effectively modeling human body trajectories. To address this limitation, this paper introduces a spatial-temporal correlation network, denoted as CorrNet+, which explicitly identifies body trajectories across multiple frames. In specific, CorrNet+ employs a correlation module and an identification module to build human body trajectories. Afterwards, a temporal attention module is followed to adaptively evaluate the contributions of different frames. The resultant features offer a holistic perspective on human body movements, facilitating a deeper understanding of sign language. As a unified model, CorrNet+ achieves new state-of-the-art performance on two extensive sign language understanding tasks, including continuous sign language recognition (CSLR) and sign language translation (SLT). Especially, CorrNet+ surpasses previous methods equipped with resource-intensive pose-estimation networks or pre-extracted heatmaps for hand and facial feature extraction. Compared with CorrNet, CorrNet+ achieves a significant performance boost across all benchmarks while halving the computational overhead. A comprehensive comparison with previous spatial-temporal reasoning methods verifies the superiority of CorrNet+. Code is available at https://github.com/hulianyuyy/CorrNet_Plus.
- 2 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 8 (6) , 3925-3935
Spatial Temporal Aggregation for Efficient Continuous Sign Language Recognition
Despite the recent progress of continuous sign language recognition (CSLR), most state-of-the-art methods process input sign language videos frame by frame to predict sentences. This usually causes a heavy computational burden and is inefficient and even infeasible in real-world scenarios. Inspired by the fact that videos are inherently redundant where not all frames are essential for recognition, we propose spatial temporal aggregation (STAgg) to address this problem. Specifically, STAgg synthesizes adjacent similar frames into a unified robust representation before being fed into the recognition module, thus highly reducing the computation complexity and memory demand. We first give a detailed analysis on commonly-used aggregation methods like subsampling, max pooling and average, and then naturally derive our STAgg from the expected design criterion. Compared to commonly used pooling and subsampling counterparts, extensive ablation studies verify the superiority of our proposed three diverse STAgg variants in both accuracy and efficiency. The best version achieves comparative accuracy with state-of-the-art competitors, but is 1.35× faster with only 0.50× computational costs, consuming 0.70× training time and 0.65× memory usage. Experiments on four large-scale datasets upon multiple backbones fully verify the generalizability and effectiveness of the proposed STAgg. Another advantage of STAgg is enabling more powerful backbones, which may further boost the accuracy of CSLR under similar computational/memory budgets. We also visualize the results of STAgg to support intuitive and insightful analysis of the effects of STAgg.
- 2 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 8 (6) , 3925-3935
Spatial Temporal Aggregation for Efficient Continuous Sign Language Recognition
Despite the recent progress of continuous sign language recognition (CSLR), most state-of-the-art methods process input sign language videos frame by frame to predict sentences. This usually causes a heavy computational burden and is inefficient and even infeasible in real-world scenarios. Inspired by the fact that videos are inherently redundant where not all frames are essential for recognition, we propose spatial temporal aggregation (STAgg) to address this problem. Specifically, STAgg synthesizes adjacent similar frames into a unified robust representation before being fed into the recognition module, thus highly reducing the computation complexity and memory demand. We first give a detailed analysis on commonly-used aggregation methods like subsampling, max pooling and average, and then naturally derive our STAgg from the expected design criterion. Compared to commonly used pooling and subsampling counterparts, extensive ablation studies verify the superiority of our proposed three diverse STAgg variants in both accuracy and efficiency. The best version achieves comparative accuracy with state-of-the-art competitors, but is 1.35× faster with only 0.50× computational costs, consuming 0.70× training time and 0.65× memory usage. Experiments on four large-scale datasets upon multiple backbones fully verify the generalizability and effectiveness of the proposed STAgg. Another advantage of STAgg is enabling more powerful backbones, which may further boost the accuracy of CSLR under similar computational/memory budgets. We also visualize the results of STAgg to support intuitive and insightful analysis of the effects of STAgg.
- Open Access
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 709-718
AdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognition
Raw videos have been proven to own considerable feature redundancy where in many cases only a portion of frames can already meet the requirements for accurate recognition. In this paper, we are interested in whether such redundancy can be effectively leveraged to facilitate efficient inference in continuous sign language recognition (CSLR). We propose a novel adaptive model (AdaBrowse) to dynamically select a most informative subsequence from input video sequences by modelling this problem as a sequential decision task. In specific, we first utilize a lightweight network to quickly scan input videos to extract coarse features. Then these features are fed into a policy network to intelligently select a subsequence to process. The corresponding subsequence is finally inferred by a normal CSLR model for sentence prediction. As only a portion of frames are processed in this procedure, the total computations can be considerably saved. Besides temporal redundancy, we are also interested in whether the inherent spatial redundancy can be seamlessly integrated together to achieve further efficiency, i.e., dynamically selecting a lowest input resolution for each sample, whose model is referred to as AdaBrowse+. Extensive experimental results on four large-scale CSLR datasets, i.e., PHOENIX14, PHOENIX14-T, CSL-Daily and CSL, demonstrate the effectiveness of AdaBrowse and AdaBrowse+ by achieving comparable accuracy with state-of-the-art methods with 1.44X throughput and 2.12X fewer FLOPs. Comparisons with other commonly-used 2D CNNs and adaptive efficient methods verify the effectiveness of AdaBrowse. Code is available at https://github.com/hulianyuyy/AdaBrowse.
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 709-718
Open AccessAdaBrowse: Adaptive Video Browser for Efficient Continuous Sign Language Recognition
Raw videos have been proven to own considerable feature redundancy where in many cases only a portion of frames can already meet the requirements for accurate recognition. In this paper, we are interested in whether such redundancy can be effectively leveraged to facilitate efficient inference in continuous sign language recognition (CSLR). We propose a novel adaptive model (AdaBrowse) to dynamically select a most informative subsequence from input video sequences by modelling this problem as a sequential decision task. In specific, we first utilize a lightweight network to quickly scan input videos to extract coarse features. Then these features are fed into a policy network to intelligently select a subsequence to process. The corresponding subsequence is finally inferred by a normal CSLR model for sentence prediction. As only a portion of frames are processed in this procedure, the total computations can be considerably saved. Besides temporal redundancy, we are also interested in whether the inherent spatial redundancy can be seamlessly integrated together to achieve further efficiency, i.e., dynamically selecting a lowest input resolution for each sample, whose model is referred to as AdaBrowse+. Extensive experimental results on four large-scale CSLR datasets, i.e., PHOENIX14, PHOENIX14-T, CSL-Daily and CSL, demonstrate the effectiveness of AdaBrowse and AdaBrowse+ by achieving comparable accuracy with state-of-the-art methods with 1.44X throughput and 2.12X fewer FLOPs. Comparisons with other commonly-used 2D CNNs and adaptive efficient methods verify the effectiveness of AdaBrowse. Code is available at https://github.com/hulianyuyy/AdaBrowse.