Jia-Hong Huang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jia-Hong Huang has a total of 47 co-authors. Here are the top co-authors by number of shared publications:
- Yixian Shen
University of Amsterdam
5 shared publications - Evangelos Kanoulas
University of Amsterdam, Amsterdam, Netherlands
4 shared publications - Hongyi Zhu
University of Amsterdam, Amsterdam, Netherlands
4 shared publications - Jesper Tegnér
King Abdullah University of Science and Technology
4 shared publications - Stevan Rudinac
University of Amsterdam, Amsterdam, Netherlands
4 shared publications - Bernard Ghanem
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
3 shared publications
Latest Publications
- 9 September 2026
- Published by Springer Nature
- p. 549-567
Keeping the Evidence Chain: Semantic Evidence Allocation for Training-Free Token Pruning in Video Temporal Grounding
- 9 September 2026
- Published by Springer Nature
- p. 549-567
Keeping the Evidence Chain: Semantic Evidence Allocation for Training-Free Token Pruning in Video Temporal Grounding
- Open Access
- 15 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 2246-2255
A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding
- 15 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 2246-2255
Open AccessA-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding
- Open Access
- 15 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 1749-1758
Agent-Based Query Reformulation: Simulating Feedback and Mitigating Negation Blindness in Interactive Image Retrieval
- 15 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 1749-1758
Open AccessAgent-Based Query Reformulation: Simulating Feedback and Mitigating Negation Blindness in Interactive Image Retrieval
- Top Cited PapersOpen Access
- 27 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 22 (3) , 1-21
Conditional Modeling-Based Automatic Video Summarization
The aim of video summarization is to shorten videos automatically while retaining the key information necessary to convey the overall story. Video summarization methods mainly rely on visual factors, such as visual consecutiveness and diversity, which may not be sufficient to fully understand the content of the video. There are other non-visual factors, such as interestingness, representativeness, and storyline consistency that should also be considered for generating high-quality video summaries. Current methods do not adequately take into account these non-visual factors, resulting in suboptimal performance. In this work, a new approach to video summarization is proposed based on insights gained from how humans create ground truth video summaries. The method utilizes a conditional modeling perspective and introduces multiple meaningful random variables and joint distributions to characterize the key components of video summarization. Helper distributions are employed to improve the training of the model. A conditional attention module is designed to mitigate potential performance degradation in the presence of multi-modal input. The proposed video summarization method incorporates the above innovative design choices that aim to narrow the gap between human-generated and machine-generated video summaries. Extensive experiments show that the proposed approach outperforms existing methods and achieves state-of-the-art performance on commonly used video summarization datasets.
- 27 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 22 (3) , 1-21
Top Cited PapersOpen AccessConditional Modeling-Based Automatic Video Summarization
The aim of video summarization is to shorten videos automatically while retaining the key information necessary to convey the overall story. Video summarization methods mainly rely on visual factors, such as visual consecutiveness and diversity, which may not be sufficient to fully understand the content of the video. There are other non-visual factors, such as interestingness, representativeness, and storyline consistency that should also be considered for generating high-quality video summaries. Current methods do not adequately take into account these non-visual factors, resulting in suboptimal performance. In this work, a new approach to video summarization is proposed based on insights gained from how humans create ground truth video summaries. The method utilizes a conditional modeling perspective and introduces multiple meaningful random variables and joint distributions to characterize the key components of video summarization. Helper distributions are employed to improve the training of the model. A conditional attention module is designed to mitigate potential performance degradation in the presence of multi-modal input. The proposed video summarization method incorporates the above innovative design choices that aim to narrow the gap between human-generated and machine-generated video summaries. Extensive experiments show that the proposed approach outperforms existing methods and achieves state-of-the-art performance on commonly used video summarization datasets.
- Open Access
- 10 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 6789-6792
DeepEyeNet: Generating Medical Report for Retinal Images
- 10 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 6789-6792
Open AccessDeepEyeNet: Generating Medical Report for Retinal Images
- Open Access
- 18 June 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
Interactive Image Retrieval Meets Query Rewriting with Large Language and Vision Language Models
Image search stands as a pivotal task in multimedia and computer vision, finding applications across diverse domains, ranging from internet search to medical diagnostics. Conventional image search systems operate by accepting textual or visual queries, retrieving the top-relevant candidate results from the database. However, prevalent methods often rely on single-turn procedures, introducing potential inaccuracies and limited recall. These methods also face the challenges, such as vocabulary mismatch and the semantic gap, constraining their overall effectiveness. To address these issues, we propose an interactive image retrieval system capable of refining queries based on user relevance feedback in a multi-turn setting. This system incorporates an image captioner based on vision language model (VLM) to enhance the quality of text-based queries, resulting in more informative queries with each iteration. Moreover, we introduce a denoiser based on large language model (LLM) to refine text-based query expansions, mitigating inaccuracies in image descriptions generated by captioning models. To evaluate our system, we curate a new dataset by adapting the MSR-VTT and MSVD video retrieval datasets to the image retrieval task, offering multiple relevant ground truth images for each query. Through comprehensive experiments, we validate the effectiveness of our proposed system against baseline methods, achieving state-of-the-art performance with a notable 10% improvement in terms of recall. Our contributions encompass the development of an innovative interactive image retrieval system, the integration of an LLM-based denoiser, the curation of a meticulously designed evaluation dataset, and thorough experimental validation.
- 18 June 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
Open AccessInteractive Image Retrieval Meets Query Rewriting with Large Language and Vision Language Models
Image search stands as a pivotal task in multimedia and computer vision, finding applications across diverse domains, ranging from internet search to medical diagnostics. Conventional image search systems operate by accepting textual or visual queries, retrieving the top-relevant candidate results from the database. However, prevalent methods often rely on single-turn procedures, introducing potential inaccuracies and limited recall. These methods also face the challenges, such as vocabulary mismatch and the semantic gap, constraining their overall effectiveness. To address these issues, we propose an interactive image retrieval system capable of refining queries based on user relevance feedback in a multi-turn setting. This system incorporates an image captioner based on vision language model (VLM) to enhance the quality of text-based queries, resulting in more informative queries with each iteration. Moreover, we introduce a denoiser based on large language model (LLM) to refine text-based query expansions, mitigating inaccuracies in image descriptions generated by captioning models. To evaluate our system, we curate a new dataset by adapting the MSR-VTT and MSVD video retrieval datasets to the image retrieval task, offering multiple relevant ground truth images for each query. Through comprehensive experiments, we validate the effectiveness of our proposed system against baseline methods, achieving state-of-the-art performance with a notable 10% improvement in terms of recall. Our contributions encompass the development of an innovative interactive image retrieval system, the integration of an LLM-based denoiser, the curation of a meticulously designed evaluation dataset, and thorough experimental validation.
- Open Access
- 21 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4554-4562
Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Models
The legal landscape encompasses a wide array of lawsuit types, presenting lawyers with challenges in delivering timely and accurate information to clients, particularly concerning critical aspects like potential imprisonment duration or financial repercussions. Compounded by the scarcity of legal experts, there's an urgent need to enhance the efficiency of traditional legal workflows. Recent advances in deep learning, especially Large Language Models (LLMs), offer promising solutions to this challenge. Leveraging LLMs' mathematical reasoning capabilities, we propose a novel approach integrating LLM-based methodologies with specially designed prompts to address precision requirements in legal Artificial Intelligence (LegalAI) applications. The proposed work seeks to bridge the gap between traditional legal practices and modern technological advancements, paving the way for a more accessible, efficient, and equitable legal system. To validate this method, we introduce a curated dataset tailored to precision-oriented LegalAI tasks, serving as a benchmark for evaluating LLM-based approaches. Extensive experimentation confirms the efficacy of our methodology in generating accurate numerical estimates within the legal domain, emphasizing the role of LLMs in streamlining legal processes and meeting the evolving demands of LegalAI. Github: https://github.com/Jhhuangkay/Optimizing-Numerical-Estimation-and-Operational-Efficiency-in-the-Legal-Domain.
- 21 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4554-4562
Open AccessOptimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Models
The legal landscape encompasses a wide array of lawsuit types, presenting lawyers with challenges in delivering timely and accurate information to clients, particularly concerning critical aspects like potential imprisonment duration or financial repercussions. Compounded by the scarcity of legal experts, there's an urgent need to enhance the efficiency of traditional legal workflows. Recent advances in deep learning, especially Large Language Models (LLMs), offer promising solutions to this challenge. Leveraging LLMs' mathematical reasoning capabilities, we propose a novel approach integrating LLM-based methodologies with specially designed prompts to address precision requirements in legal Artificial Intelligence (LegalAI) applications. The proposed work seeks to bridge the gap between traditional legal practices and modern technological advancements, paving the way for a more accessible, efficient, and equitable legal system. To validate this method, we introduce a curated dataset tailored to precision-oriented LegalAI tasks, serving as a benchmark for evaluating LLM-based approaches. Extensive experimentation confirms the efficacy of our methodology in generating accurate numerical estimates within the legal domain, emphasizing the role of LLMs in streamlining legal processes and meeting the evolving demands of LegalAI. Github: https://github.com/Jhhuangkay/Optimizing-Numerical-Estimation-and-Operational-Efficiency-in-the-Legal-Domain.
- Open Access
- 30 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1214-1218
Multi-modal Video Summarization
The increasing prevalence of video content on platforms like YouTube and Vimeo has led to a growing demand for effective video summarization techniques. These methods aim to extract relevant content from videos and present it in a condensed form, addressing the challenge of information overload faced by users. However, the quality of the generated summaries is crucial, especially in fields like forensics, journalism, and sports analysis, where accurate and concise summaries are essential for decision-making and analysis. Traditional video summarization methods often rely solely on visual information, limiting their ability to capture textual cues and emotional content present in videos. To overcome this limitation, this thesis proposes leveraging text-based queries as context to enhance the effectiveness of video summarization. By incorporating textual information alongside visual data, the proposed approach aims to generate query-dependent video summaries that better align with users' requirements. Furthermore, the thesis leverages a novel conditional modeling perspective to impart a more human-like quality to the video summarization process. These methods hold promise for various applications, including documentary filmmaking and educational content creation. Additionally, the thesis addresses the challenge of data scarcity in video summarization by proposing a self-supervised learning approach that leverages pretext tasks to generate pseudo-labels for model training. In 2020, we proposed a query-controllable video summarization technology, which is elaborated upon in the second chapter of this thesis. Notably, Google has adopted a similar feature, integrating it into their large language model (LLM) Gemini Pro-1.5 version, in 2024.
- 30 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1214-1218
Open AccessMulti-modal Video Summarization
The increasing prevalence of video content on platforms like YouTube and Vimeo has led to a growing demand for effective video summarization techniques. These methods aim to extract relevant content from videos and present it in a condensed form, addressing the challenge of information overload faced by users. However, the quality of the generated summaries is crucial, especially in fields like forensics, journalism, and sports analysis, where accurate and concise summaries are essential for decision-making and analysis. Traditional video summarization methods often rely solely on visual information, limiting their ability to capture textual cues and emotional content present in videos. To overcome this limitation, this thesis proposes leveraging text-based queries as context to enhance the effectiveness of video summarization. By incorporating textual information alongside visual data, the proposed approach aims to generate query-dependent video summaries that better align with users' requirements. Furthermore, the thesis leverages a novel conditional modeling perspective to impart a more human-like quality to the video summarization process. These methods hold promise for various applications, including documentary filmmaking and educational content creation. Additionally, the thesis addresses the challenge of data scarcity in video summarization by proposing a self-supervised learning approach that leverages pretext tasks to generate pseudo-labels for model training. In 2020, we proposed a query-controllable video summarization technology, which is elaborated upon in the second chapter of this thesis. Notably, Google has adopted a similar feature, integrating it into their large language model (LLM) Gemini Pro-1.5 version, in 2024.
- Open Access
- 30 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 978-987
Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models
Image search stands as a pivotal task in multimedia and computer vision, finding applications across diverse domains, ranging from internet search to medical diagnostics. Conventional image search systems operate by accepting textual or visual queries, retrieving the top-relevant candidate results from the database. However, prevalent methods often rely on single-turn procedures, introducing potential inaccuracies and limited recall. These methods also face the challenges, such as vocabulary mismatch and the semantic gap, constraining their overall effectiveness. To address these issues, we propose an interactive image retrieval system capable of refining queries based on user relevance feedback in a multi-turn setting. This system incorporates a vision language model (VLM) based image captioner to enhance the quality of text-based queries, resulting in more informative queries with each iteration. Moreover, we introduce a large language model (LLM) based denoiser to refine text-based query expansions, mitigating inaccuracies in image descriptions generated by captioning models. To evaluate our system, we curate a new dataset by adapting the MSR-VTT video retrieval dataset to the image retrieval task, offering multiple relevant ground truth images for each query. Through comprehensive experiments, we validate the effectiveness of our proposed system against baseline methods, achieving state-of-the-art performance with a notable 10% improvement in terms of recall. Our contributions encompass the development of an innovative interactive image retrieval system, the integration of an LLM-based denoiser, the curation of a meticulously designed evaluation dataset, and thorough experimental validation.
- 30 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 978-987
Open AccessEnhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models
Image search stands as a pivotal task in multimedia and computer vision, finding applications across diverse domains, ranging from internet search to medical diagnostics. Conventional image search systems operate by accepting textual or visual queries, retrieving the top-relevant candidate results from the database. However, prevalent methods often rely on single-turn procedures, introducing potential inaccuracies and limited recall. These methods also face the challenges, such as vocabulary mismatch and the semantic gap, constraining their overall effectiveness. To address these issues, we propose an interactive image retrieval system capable of refining queries based on user relevance feedback in a multi-turn setting. This system incorporates a vision language model (VLM) based image captioner to enhance the quality of text-based queries, resulting in more informative queries with each iteration. Moreover, we introduce a large language model (LLM) based denoiser to refine text-based query expansions, mitigating inaccuracies in image descriptions generated by captioning models. To evaluate our system, we curate a new dataset by adapting the MSR-VTT video retrieval dataset to the image retrieval task, offering multiple relevant ground truth images for each query. Through comprehensive experiments, we validate the effectiveness of our proposed system against baseline methods, achieving state-of-the-art performance with a notable 10% improvement in terms of recall. Our contributions encompass the development of an innovative interactive image retrieval system, the integration of an LLM-based denoiser, the curation of a meticulously designed evaluation dataset, and thorough experimental validation.