HSHumphrey Shi

Humphrey Shi

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Humphrey Shi has a total of 109 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Beyond Realism: Learning the Art of Expressive Composition with StickerNet

    by Haoming Lu, David Kocharian, Humphrey Shi

    As a widely used operation in image editing workflows, image composition has traditionally been studied with a focus on achieving visual realism and semantic plausibility. However, in practical editing scenarios of the modern content creation landscape, many compositions are not intended to preserve realism. Instead, users of online platforms motivated by gaining community recognition often aim to create content that is more artistic, playful, or socially engaging. Taking inspiration from this observation, we define the expressive composition task, a new formulation of image composition that embraces stylistic diversity and looser placement logic, reflecting how users edit images on real-world creative platforms. To address this underexplored problem, we present StickerNet, a two-stage framework that first determines the composition type, then predicts placement parameters such as opacity, mask, location, and scale accordingly. Unlike prior work that constructs datasets by simulating object placements on real images, we directly build our dataset from 1.8 million editing actions collected on an anonymous online visual creation and editing platform, each reflecting user-community validated placement decisions. This grounding in authentic editing behavior ensures strong alignment between task definition and training supervision. User studies and quantitative evaluations show that StickerNet outperforms common baselines and closely matches human placement behavior, demonstrating the effectiveness of learning from real-world editing patterns despite the inherent ambiguity of the task. This work introduces a new direction in visual understanding that emphasizes expressiveness and user intent over realism.

  • GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Models

    by Moreno D’iNcà, Elia Peruzzo, Massimiliano Mancini, Xingqian Xu, Humphrey Shi, Nicu Sebe

    Recent progress in Text-to-Image (T2I) generative models has enabled high-quality image generation. As performance and accessibility increase, these models are gaining significant attraction and popularity: ensuring their fairness and safety is a priority to prevent the dissemination and perpetuation of biases. However, existing studies in bias detection focus on closed sets of predefined biases (e.g., gender, ethnicity). In this paper, we propose a general framework to identify, quantify, and explain biases in an open set setting, i.e.,without requiring a predefined set. This pipeline leverages a Large Language Model (LLM) to propose biases starting from a set of captions. Next, these captions are used by the target generative model for generating a set of images. Finally, Vision Question Answering (VQA) is leveraged for bias evaluation. We show two variations of this framework: OpenBias and GradBias. OpenBias detects and quantifies biases, while GradBias determines the contribution of individual prompt words on biases. OpenBias effectively detects both well-known and novel biases related to people, objects, and animals and highly aligns with existing closed-set bias detection methods and human judgment. GradBias shows that neutral words can significantly influence biases and it outperforms several baselines, including state-of-the-art foundation models. Code available here: https://github.com/Moreno98/GradBias.


  • Learning Trimaps via Clicks for Image Matting

    by Chenyi Zhang, Yihan Hu, Henghui Ding, Humphrey Shi, Yao Zhao, Yunchao Wei

    Despite the significant advancements achieved in image matting, the existing models heavily depend on manually drawn trimaps to produce accurate results in natural image scenarios. However, the process of obtaining trimaps is time-consuming and lacks user-friendliness and device compatibility. This greatly limits the practical applicability of all trimap-based matting methods. To address this issue, we introduce Click2Trimap, an interactive model that is capable of predicting high-quality trimaps and alpha mattes with minimal user click inputs. By analyzing real users' behavioral logic and the characteristics of trimaps, we successfully propose a powerful iterative three-class training strategy and a dedicated simulation function, making Click2Trimap exhibit versatility across various scenarios. Compared with all existing trimap-free matting methods, Click2Trimap achieves superior performance in quantitative and qualitative assessments conducted on synthetic and real-world matting datasets. In particular, in a user study, Click2Trimap yields high-quality trimap and matting predictions in just 5 seconds per image on average, demonstrating its substantial practical value for use in real-world applications.


  • Open Access

    R ag M e : Retrieval Augmented Video Generation for Enhanced Motion Realism

    by Elia Peruzzo, Dejia Xu, Xingqian Xu, Humphrey Shi, Nicu Sebe


  • Open Access

    CompactFlowNet: Efficient Real-time Optical Flow Estimation on Mobile Devices

    by Andrei Znobishchev, Valerii Filev, Oleg Kudashev, Nikita Orlov, Humphrey Shi

    We present CompactFlowNet, the first real-time mobile neural network for optical flow prediction, which involves determining the displacement of each pixel in an initial frame relative to the corresponding pixel in a subsequent frame. Optical flow serves as a fundamental building block for various video-related tasks, such as video restoration, motion estimation, video stabilization, object tracking, action recognition, and video generation. While current state-of-the-art methods prioritize accuracy, they often overlook constraints regarding speed and memory usage. Existing light models typically focus on reducing size but still exhibit high latency, compromise significantly on quality, or are optimized for high-performance GPUs, resulting in sub-optimal performance on mobile devices. This study aims to develop a mobile-optimized optical flow model by proposing a novel mobile device-compatible architecture, as well as enhancements to the training pipeline, which optimize the model for reduced weight, low memory utilization, and increased speed while maintaining minimal error. Our approach demonstrates superior or comparable performance to the state-of-the-art lightweight models on the challenging KITTI and Sintel benchmarks. Furthermore, it attains a significantly accelerated inference speed, thereby yielding real-time operational efficiency on the iPhone 8, while surpassing real-time performance levels on more advanced mobile devices.

  • Open Access

    Towards Responsible Use of Large Multi-modal AI to Analyze Human Social Behaviors

    by Qingxiao Zheng, Xi Lu, Qiao Jin, Jitesh Jain, Hedda Meadan-Kaplansky, Humphrey Shi, Jinjun Xiong, Yun Huang

    This SIG is primarily designed for researchers and practitioners interested in exploring the utilization of multimodal artificial intelligence (AI) to understand and analyze human behaviors across various domains. It will particularly benefit professionals in UX design and research, cognitive science, education, psychology, and data science who are pursuing innovative ways to integrate multimodal data with AI to gain deeper behavioral insights. The session will focus on using AI to track, visualize, and generate insights around both verbal and non-verbal social interactions for various purposes, such as video question-answering. Discussions will cover practical applications in settings such as online learning environments, telecommunication, and social media platforms. This SIG will also provoke reflection on the ethical stewardship of AI, highlighting the importance of responsible usage to mitigate concerns such as algorithmic bias and data privacy.


  • Open Access

    UVMap-ID: A Controllable and Personalized UV Map Generative Model

    by Weijie Wang, Jichao Zhang, Chang Liu, Xia Li, Xingqian Xu, Humphrey Shi, Nicu Sebe, Bruno Lepri

    Recently, diffusion models have made significant strides in synthesizing realistic 2D human images based on provided text prompts. Building upon this, researchers have extended 2D text-to-image diffusion models into the 3D domain for generating human textures (UV Maps). However, some important problems about UV Map Generative models are still not solved, i.e., how to generate personalized texture maps for any given face image, and how to define and evaluate the quality of these generated texture maps. To solve the above problems, we introduce a novel method, UVMap-ID, which is a controllable and personalized UV Map generative model. Unlike traditional large-scale training methods in 2D, we propose to fine-tune a pre-trained text-to-image diffusion model which is integrated with a face fusion module for achieving ID-driven customized generation. To support the finetuning strategy, we introduce a small-scale attribute-balanced training dataset, including high-quality textures with labeled text and Face ID. Additionally, we introduce some metrics to evaluate the multiple aspects of the textures. Finally, both quantitative and qualitative analyses demonstrate the effectiveness of our method in controllable and personalized UV Map generation.


  • Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation

    by Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yunchao Wei, Humphrey Shi


  • Diffusion for Natural Image Matting

    by Yihan Hu, Yiheng Lin, Wei Wang, Yao Zhao, Yunchao Wei, Humphrey Shi