Humphrey Shi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Humphrey Shi has a total of 109 co-authors. Here are the top co-authors by number of shared publications:
- Yunchao Wei
Beijing Jiaotong University, Beijing, China
7 shared publications - Gao Huang
Tsinghua University, Beijing, China
5 shared publications - Zhangyang Wang
The University of Texas at Austin, Austin, USA
5 shared publications - Nicu Sebe
University of Trento, Trento, Italy
4 shared publications - Yao Zhao
Institute of Information Science, Beijing Jiaotong University, Beijing, China and Visual Intelligence + X International Joint Laboratory, Beijing, China
4 shared publications - Yi Yang
Zhejiang University, China
4 shared publications
Latest Publications
- Open Access
Beyond Realism: Learning the Art of Expressive Composition with StickerNet
As a widely used operation in image editing workflows, image composition has traditionally been studied with a focus on achieving visual realism and semantic plausibility. However, in practical editing scenarios of the modern content creation landscape, many compositions are not intended to preserve realism. Instead, users of online platforms motivated by gaining community recognition often aim to create content that is more artistic, playful, or socially engaging. Taking inspiration from this observation, we define the expressive composition task, a new formulation of image composition that embraces stylistic diversity and looser placement logic, reflecting how users edit images on real-world creative platforms. To address this underexplored problem, we present StickerNet, a two-stage framework that first determines the composition type, then predicts placement parameters such as opacity, mask, location, and scale accordingly. Unlike prior work that constructs datasets by simulating object placements on real images, we directly build our dataset from 1.8 million editing actions collected on an anonymous online visual creation and editing platform, each reflecting user-community validated placement decisions. This grounding in authentic editing behavior ensures strong alignment between task definition and training supervision. User studies and quantitative evaluations show that StickerNet outperforms common baselines and closely matches human placement behavior, demonstrating the effectiveness of learning from real-world editing patterns despite the inherent ambiguity of the task. This work introduces a new direction in visual understanding that emphasizes expressiveness and user intent over realism.
Open AccessBeyond Realism: Learning the Art of Expressive Composition with StickerNet
As a widely used operation in image editing workflows, image composition has traditionally been studied with a focus on achieving visual realism and semantic plausibility. However, in practical editing scenarios of the modern content creation landscape, many compositions are not intended to preserve realism. Instead, users of online platforms motivated by gaining community recognition often aim to create content that is more artistic, playful, or socially engaging. Taking inspiration from this observation, we define the expressive composition task, a new formulation of image composition that embraces stylistic diversity and looser placement logic, reflecting how users edit images on real-world creative platforms. To address this underexplored problem, we present StickerNet, a two-stage framework that first determines the composition type, then predicts placement parameters such as opacity, mask, location, and scale accordingly. Unlike prior work that constructs datasets by simulating object placements on real images, we directly build our dataset from 1.8 million editing actions collected on an anonymous online visual creation and editing platform, each reflecting user-community validated placement decisions. This grounding in authentic editing behavior ensures strong alignment between task definition and training supervision. User studies and quantitative evaluations show that StickerNet outperforms common baselines and closely matches human placement behavior, demonstrating the effectiveness of learning from real-world editing patterns despite the inherent ambiguity of the task. This work introduces a new direction in visual understanding that emphasizes expressiveness and user intent over realism.
- 1 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (11) , 9863-9875
GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Models
Recent progress in Text-to-Image (T2I) generative models has enabled high-quality image generation. As performance and accessibility increase, these models are gaining significant attraction and popularity: ensuring their fairness and safety is a priority to prevent the dissemination and perpetuation of biases. However, existing studies in bias detection focus on closed sets of predefined biases (e.g., gender, ethnicity). In this paper, we propose a general framework to identify, quantify, and explain biases in an open set setting, i.e.,without requiring a predefined set. This pipeline leverages a Large Language Model (LLM) to propose biases starting from a set of captions. Next, these captions are used by the target generative model for generating a set of images. Finally, Vision Question Answering (VQA) is leveraged for bias evaluation. We show two variations of this framework: OpenBias and GradBias. OpenBias detects and quantifies biases, while GradBias determines the contribution of individual prompt words on biases. OpenBias effectively detects both well-known and novel biases related to people, objects, and animals and highly aligns with existing closed-set bias detection methods and human judgment. GradBias shows that neutral words can significantly influence biases and it outperforms several baselines, including state-of-the-art foundation models. Code available here: https://github.com/Moreno98/GradBias.
- 1 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (11) , 9863-9875
GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Models
Recent progress in Text-to-Image (T2I) generative models has enabled high-quality image generation. As performance and accessibility increase, these models are gaining significant attraction and popularity: ensuring their fairness and safety is a priority to prevent the dissemination and perpetuation of biases. However, existing studies in bias detection focus on closed sets of predefined biases (e.g., gender, ethnicity). In this paper, we propose a general framework to identify, quantify, and explain biases in an open set setting, i.e.,without requiring a predefined set. This pipeline leverages a Large Language Model (LLM) to propose biases starting from a set of captions. Next, these captions are used by the target generative model for generating a set of images. Finally, Vision Question Answering (VQA) is leveraged for bias evaluation. We show two variations of this framework: OpenBias and GradBias. OpenBias detects and quantifies biases, while GradBias determines the contribution of individual prompt words on biases. OpenBias effectively detects both well-known and novel biases related to people, objects, and animals and highly aligns with existing closed-set bias detection methods and human judgment. GradBias shows that neutral words can significantly influence biases and it outperforms several baselines, including state-of-the-art foundation models. Code available here: https://github.com/Moreno98/GradBias.
- 9 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 8668-8678
Learning Trimaps via Clicks for Image Matting
Despite the significant advancements achieved in image matting, the existing models heavily depend on manually drawn trimaps to produce accurate results in natural image scenarios. However, the process of obtaining trimaps is time-consuming and lacks user-friendliness and device compatibility. This greatly limits the practical applicability of all trimap-based matting methods. To address this issue, we introduce Click2Trimap, an interactive model that is capable of predicting high-quality trimaps and alpha mattes with minimal user click inputs. By analyzing real users' behavioral logic and the characteristics of trimaps, we successfully propose a powerful iterative three-class training strategy and a dedicated simulation function, making Click2Trimap exhibit versatility across various scenarios. Compared with all existing trimap-free matting methods, Click2Trimap achieves superior performance in quantitative and qualitative assessments conducted on synthetic and real-world matting datasets. In particular, in a user study, Click2Trimap yields high-quality trimap and matting predictions in just 5 seconds per image on average, demonstrating its substantial practical value for use in real-world applications.
- 9 September 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 8668-8678
Learning Trimaps via Clicks for Image Matting
Despite the significant advancements achieved in image matting, the existing models heavily depend on manually drawn trimaps to produce accurate results in natural image scenarios. However, the process of obtaining trimaps is time-consuming and lacks user-friendliness and device compatibility. This greatly limits the practical applicability of all trimap-based matting methods. To address this issue, we introduce Click2Trimap, an interactive model that is capable of predicting high-quality trimaps and alpha mattes with minimal user click inputs. By analyzing real users' behavioral logic and the characteristics of trimaps, we successfully propose a powerful iterative three-class training strategy and a dedicated simulation function, making Click2Trimap exhibit versatility across various scenarios. Compared with all existing trimap-free matting methods, Click2Trimap achieves superior performance in quantitative and qualitative assessments conducted on synthetic and real-world matting datasets. In particular, in a user study, Click2Trimap yields high-quality trimap and matting predictions in just 5 seconds per image on average, demonstrating its substantial practical value for use in real-world applications.
- Open Access
- 30 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 1081-1090
R ag M e : Retrieval Augmented Video Generation for Enhanced Motion Realism
- 30 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 1081-1090
Open AccessR ag M e : Retrieval Augmented Video Generation for Enhanced Motion Realism
- 27 December 2024
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 133 (6) , 3166-3179
Combating Label Noise with a General Surrogate Model for Sample Selection
- 27 December 2024
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 133 (6) , 3166-3179
Combating Label Noise with a General Surrogate Model for Sample Selection
- Open Access
CompactFlowNet: Efficient Real-time Optical Flow Estimation on Mobile Devices
We present CompactFlowNet, the first real-time mobile neural network for optical flow prediction, which involves determining the displacement of each pixel in an initial frame relative to the corresponding pixel in a subsequent frame. Optical flow serves as a fundamental building block for various video-related tasks, such as video restoration, motion estimation, video stabilization, object tracking, action recognition, and video generation. While current state-of-the-art methods prioritize accuracy, they often overlook constraints regarding speed and memory usage. Existing light models typically focus on reducing size but still exhibit high latency, compromise significantly on quality, or are optimized for high-performance GPUs, resulting in sub-optimal performance on mobile devices. This study aims to develop a mobile-optimized optical flow model by proposing a novel mobile device-compatible architecture, as well as enhancements to the training pipeline, which optimize the model for reduced weight, low memory utilization, and increased speed while maintaining minimal error. Our approach demonstrates superior or comparable performance to the state-of-the-art lightweight models on the challenging KITTI and Sintel benchmarks. Furthermore, it attains a significantly accelerated inference speed, thereby yielding real-time operational efficiency on the iPhone 8, while surpassing real-time performance levels on more advanced mobile devices.
Open AccessCompactFlowNet: Efficient Real-time Optical Flow Estimation on Mobile Devices
We present CompactFlowNet, the first real-time mobile neural network for optical flow prediction, which involves determining the displacement of each pixel in an initial frame relative to the corresponding pixel in a subsequent frame. Optical flow serves as a fundamental building block for various video-related tasks, such as video restoration, motion estimation, video stabilization, object tracking, action recognition, and video generation. While current state-of-the-art methods prioritize accuracy, they often overlook constraints regarding speed and memory usage. Existing light models typically focus on reducing size but still exhibit high latency, compromise significantly on quality, or are optimized for high-performance GPUs, resulting in sub-optimal performance on mobile devices. This study aims to develop a mobile-optimized optical flow model by proposing a novel mobile device-compatible architecture, as well as enhancements to the training pipeline, which optimize the model for reduced weight, low memory utilization, and increased speed while maintaining minimal error. Our approach demonstrates superior or comparable performance to the state-of-the-art lightweight models on the challenging KITTI and Sintel benchmarks. Furthermore, it attains a significantly accelerated inference speed, thereby yielding real-time operational efficiency on the iPhone 8, while surpassing real-time performance levels on more advanced mobile devices.
- Open Access
- 11 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 663-665
Towards Responsible Use of Large Multi-modal AI to Analyze Human Social Behaviors
This SIG is primarily designed for researchers and practitioners interested in exploring the utilization of multimodal artificial intelligence (AI) to understand and analyze human behaviors across various domains. It will particularly benefit professionals in UX design and research, cognitive science, education, psychology, and data science who are pursuing innovative ways to integrate multimodal data with AI to gain deeper behavioral insights. The session will focus on using AI to track, visualize, and generate insights around both verbal and non-verbal social interactions for various purposes, such as video question-answering. Discussions will cover practical applications in settings such as online learning environments, telecommunication, and social media platforms. This SIG will also provoke reflection on the ethical stewardship of AI, highlighting the importance of responsible usage to mitigate concerns such as algorithmic bias and data privacy.
- 11 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 663-665
Open AccessTowards Responsible Use of Large Multi-modal AI to Analyze Human Social Behaviors
This SIG is primarily designed for researchers and practitioners interested in exploring the utilization of multimodal artificial intelligence (AI) to understand and analyze human behaviors across various domains. It will particularly benefit professionals in UX design and research, cognitive science, education, psychology, and data science who are pursuing innovative ways to integrate multimodal data with AI to gain deeper behavioral insights. The session will focus on using AI to track, visualize, and generate insights around both verbal and non-verbal social interactions for various purposes, such as video question-answering. Discussions will cover practical applications in settings such as online learning environments, telecommunication, and social media platforms. This SIG will also provoke reflection on the ethical stewardship of AI, highlighting the importance of responsible usage to mitigate concerns such as algorithmic bias and data privacy.
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 10725-10734
UVMap-ID: A Controllable and Personalized UV Map Generative Model
Recently, diffusion models have made significant strides in synthesizing realistic 2D human images based on provided text prompts. Building upon this, researchers have extended 2D text-to-image diffusion models into the 3D domain for generating human textures (UV Maps). However, some important problems about UV Map Generative models are still not solved, i.e., how to generate personalized texture maps for any given face image, and how to define and evaluate the quality of these generated texture maps. To solve the above problems, we introduce a novel method, UVMap-ID, which is a controllable and personalized UV Map generative model. Unlike traditional large-scale training methods in 2D, we propose to fine-tune a pre-trained text-to-image diffusion model which is integrated with a face fusion module for achieving ID-driven customized generation. To support the finetuning strategy, we introduce a small-scale attribute-balanced training dataset, including high-quality textures with labeled text and Face ID. Additionally, we introduce some metrics to evaluate the multiple aspects of the textures. Finally, both quantitative and qualitative analyses demonstrate the effectiveness of our method in controllable and personalized UV Map generation.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 10725-10734
Open AccessUVMap-ID: A Controllable and Personalized UV Map Generative Model
Recently, diffusion models have made significant strides in synthesizing realistic 2D human images based on provided text prompts. Building upon this, researchers have extended 2D text-to-image diffusion models into the 3D domain for generating human textures (UV Maps). However, some important problems about UV Map Generative models are still not solved, i.e., how to generate personalized texture maps for any given face image, and how to define and evaluate the quality of these generated texture maps. To solve the above problems, we introduce a novel method, UVMap-ID, which is a controllable and personalized UV Map generative model. Unlike traditional large-scale training methods in 2D, we propose to fine-tune a pre-trained text-to-image diffusion model which is integrated with a face fusion module for achieving ID-driven customized generation. To support the finetuning strategy, we introduce a small-scale attribute-balanced training dataset, including high-quality textures with labeled text and Face ID. Additionally, we introduce some metrics to evaluate the multiple aspects of the textures. Finally, both quantitative and qualitative analyses demonstrate the effectiveness of our method in controllable and personalized UV Map generation.
- 25 October 2024
- Published by Springer Nature
- p. 399-416
Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation
- 25 October 2024
- Published by Springer Nature
- p. 399-416
Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation