WHWenbin He

Wenbin He

*This is an algorithmically generated author record based on publicly available information.

Wenbin He has a total of 73 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • I n F i C on D: Investigating Interactive No‐code Fine‐tuning with Concept‐based Knowledge Distillation

    by J. Huang, W. He, L. Gou, L. Ren, C. Bryan

    Knowledge distillation is a widely used technique whereby knowledge from large pre‐trained models is transferred into smaller student models. However, this process is non‐trivial and traditionally requires technical and theoretical expertise in AI/ML. We investigate a visualization‐driven strategy for making this process more accessible and intuitive via developing I n F i C on D, a novel tool that leverages visual concepts to scaffold the knowledge distillation process and support subsequent no‐code fine‐tuning of student models. I n F i C on D's backend pipeline extracts text‐aligned visual concepts and constructs highly interpretable student models; its frontend supports interactively fine‐tuning these student models by directly manipulating concept influences. Empirical evaluations help validate that I n F i C on D effectively supports knowledge distillation and subsequent fine‐tuning workflows. We additionally discuss insights and lessons learned about how human‐in‐the‐loop and visualization‐driven approaches like I n F i C on D can support accessible and adaptable AI explainability and model distillation.

  • Open Access

    Demonstration of W ay P oint : Interactive Natural Language Querying for Spatio-Temporal Video Events

    by Taehyeok Jang, Jie Jeff Xu, Suraj Srinivas, Jorge Piazentin Ono, Wenbin He, Liu Ren, Kexin Rong

    Natural language (NL) provides an intuitive interface for querying video datasets. However, existing approaches that use Multimodal Large Language Models to directly map NL queries to video results offer limited feedback mechanisms beyond prompt engineering. We present WayPoint, a prototype system for NL-based querying of spatio-temporal video events through a user-in-the-loop approach. WayPoint translates NL queries into a keyframe-inspired intermediate representation that enables fuzzy matching of spatio-temporal constraints. This structured representation allows users to refine query semantics through natural language feedback that is applied to the query logic and has deterministic effects on execution. Users can also tune retrieval performance by providing feedback on retrieved results. We demonstrate WayPoint's query synthesis and retrieval refinement workflows on the nuScenes dataset.

  • Open Access

    Scaling Expert Feedback with Reflective Edit Propagation in Compositional Knowledge Bases

    by Jiajing Guo, Xueming Li, Jorge H Piazentin Ono, Wenbin He, Liu Ren

  • InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions

    by Juntong Chen, Jiang Wu, Jiajing Guo, Vikram Mohanty, Xueming Li, Jorge Piazentin Ono, Wenbin He, Liu Ren, Dongyu Liu

    The rise of Large Language Models (LLMs) and generative visual analytics systems has transformed data‐driven insights, yet significant challenges persist in accurately interpreting users analytical and interaction intents. While language inputs offer flexibility, they often lack precision, making the expression of complex intents inefficient, error‐prone, and time‐intensive. To address these limitations, we investigate the design space of multimodal interactions for generative visual analytics through a literature review and pilot brainstorming sessions. Building on these insights, we introduce a highly extensible workflow that integrates multiple LLM agents for intent inference and visualization generation. We develop InterChat, a generative visual analytics system that combines direct manipulation of visual elements with natural language inputs. This integration enables precise intent communication and supports progressive, visually driven exploratory data analyses. By employing effective prompt engineering, and contextual interaction linking, alongside intuitive visualization and interaction designs, InterChat bridges the gap between user interactions and LLM‐driven visualizations, enhancing both interpretability and usability. Extensive evaluations, including two usage scenarios, a user study, and expert feedback, demonstrate the effectiveness of InterChat. Results show significant improvements in the accuracy and efficiency of handling complex visual analytics tasks, highlighting the potential of multimodal interactions to redefine user engagement and analytical depth in generative visual analytics.


  • VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labels

    by Xiwei Xuan, Xiaoqi Wang, Wenbin He, Jorge Piazentin Ono, Liang Gou, Kwan-Liu Ma, Liu Ren

    The advances in multi-modal foundation models (FMs) (e.g., CLIP and LLaVA) have facilitated the auto-labeling of large-scale datasets, enhancing model performance in challenging downstream tasks such as open-vocabulary object detection and segmentation. However, the quality of FM-generated labels is less studied as existing approaches focus more on data quantity over quality. This is because validating large volumes of data without ground truth presents a considerable challenge in practice. Existing methods typically rely on limited metrics to identify problematic data, lacking a comprehensive perspective, or apply human validation to only a small data fraction, failing to address the full spectrum of potential issues. To overcome these challenges, we introduce VISTA, a visual analytics framework that improves data quality to enhance the performance of multi-modal models. Targeting the complex and demanding domain of open-vocabulary image segmentation, VISTA integrates multi-phased data validation strategies with human expertise, enabling humans to identify, understand, and correct hidden issues within FM-generated labels. Through detailed use cases on two benchmark datasets and expert reviews, we demonstrate VISTA’s effectiveness from both quantitative and qualitative perspectives.


  • MetaAT: Active Testing for Label-Efficient Evaluation of Dense Recognition Tasks

    by Sanbao Su, Xin Li, Thang Doan, Sima Behpour, Wenbin He, Liang Gou, Fei Miao, Liu Ren

  • Open Access

    Demonstration of VCR: A Tabular Data Slicing Approach to Understanding Object Detection Model Performance

    by Jie Jeff Xu, Saahir Dhanani, Jorge Piazentin Ono, Wenbin He, Liu Ren, Kexin Rong

    In this demonstration, we present VCR, an automated slice discovery method (SDM) for object detection models that helps practitioners identify and explain specific scenarios in which their models exhibit systematic errors. VCR leverages the capabilities of vision foundation models to generate segment-level visual concepts that serve as interpretable explanation primitives. By integrating these visual concepts with additional image metadata in a tabular format, VCR uses a scalable frequent itemset mining-based technique to identify common patterns associated with model performance. We will demonstrate VCR's capabilities through three usage scenarios. First, users can explore the automatically extracted visual concepts and their associated labels. Second, users can run slice finding on a large object detection dataset and visually inspect the results to discover systematic errors. Finally, users can iteratively refine their slicing results by providing feedback on the granularity of visual concepts and the quality of the generated labels. These scenarios will illustrate how VCR can aid practitioners in discovering non-trivial gaps in their models' performance, providing actionable insights for model improvement.


  • Open Access

    Hyp-OW: Exploiting Hierarchical Structure Learning with Hyperbolic Distance Enhances Open World Object Detection

    by Thang Doan, Xin Li, Sima Behpour, Wenbin He, Liang Gou, Liu Ren

    Open World Object Detection (OWOD) is a challenging and realistic task that extends beyond the scope of standard Object Detection task. It involves detecting both known and unknown objects while integrating learned knowledge for future tasks. However, the level of "unknownness" varies significantly depending on the context. For example, a tree is typically considered part of the background in a self-driving scene, but it may be significant in a household context. We argue that this contextual information should already be embedded within the known classes. In other words, there should be a semantic or latent structure relationship between the known and unknown items to be discovered. Motivated by this observation, we propose Hyp-OW, a method that learns and models hierarchical representation of known items through a SuperClass Regularizer. Leveraging this representation allows us to effectively detect unknown objects using a similarity distance-based relabeling module. Extensive experiments on benchmark datasets demonstrate the effectiveness of Hyp-OW, achieving improvement in both known and unknown detection (up to 6 percent). These findings are particularly pronounced in our newly designed benchmark, where a strong hierarchical structure exists between known and unknown objects.


  • Open Access

    Slicing, Chatting, and Refining: A Concept-Based Approach for Machine Learning Model Validation with ConceptSlicer

    by Xiaoyu Zhang, Jorge H Piazentin Ono, Wenbin He, Liang Gou, Mrinmaya Sachan, Kwan-Liu Ma, Liu Ren

    As machine learning (ML) gains wider adoption in real-world applications, the validation of ML models becomes fundamental for its productization, particularly in safety-critical applications. Recently, data slice finding has emerged as a popular method for validating ML models, but it requires additional metadata or cross-modal embeddings for the slices to be interpretable. We propose ConceptSlicer, an integrated workflow that facilitates the slicing of computer vision models using visual concepts. This approach breaks down the image dataset into interpretable visual concepts, serving as metadata in the slice finding process. Our system offers insights into model issues and enables a deeper understanding of computer vision models’ strengths and weaknesses. We evaluate ConceptSlicer through interviews with eight domain experts and machine learning practitioners, and fine-tune the ML models based on their feedback. Our study also highlights varied attitudes towards large foundational models, encouraging contemplation of the challenges and opportunities presented by this technological advancement.


  • Open Access

    Hyp-OW: Exploiting Hierarchical Structure Learning with Hyperbolic Distance Enhances Open World Object Detection

    by Thang Doan, Xin Li, Sima Behpour, Wenbin He, Liang Gou, Liu Ren

    Open World Object Detection (OWOD) is a challenging and realistic task that extends beyond the scope of standard Object Detection task. It involves detecting both known and unknown objects while integrating learned knowledge for future tasks. However, the level of "unknownness" varies significantly depending on the context. For example, a tree is typically considered part of the background in a self-driving scene, but it may be significant in a household context. We argue that this contextual information should already be embedded within the known classes. In other words, there should be a semantic or latent structure relationship between the known and unknown items to be discovered. Motivated by this observation, we propose Hyp-OW, a method that learns and models hierarchical representation of known items through a SuperClass Regularizer. Leveraging this representation allows us to effectively detect unknown objects using a similarity distance-based relabeling module. Extensive experiments on benchmark datasets demonstrate the effectiveness of Hyp-OW, achieving improvement in both known and unknown detection (up to 6 percent). These findings are particularly pronounced in our newly designed benchmark, where a strong hierarchical structure exists between known and unknown objects. Our code can be found at https://github.com/boschresearch/Hyp-OW