Zhipeng Jin
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Zhipeng Jin has a total of 20 co-authors. Here are the top co-authors by number of shared publications:
- Lin Liu
Baidu Inc., Beijing, China
7 shared publications - Shuanglong Li
Baidu Inc., Beijing, China
7 shared publications - Yi Yang
Baidu Inc., Beijing, China
7 shared publications - Wen Tao
Baidu Inc., Beijing, China
6 shared publications - Cong Han
Baidu Inc., Beijing, China
5 shared publications - Kang Zhao
Baidu Search Ads, Baidu Inc., Beijing, China
4 shared publications
Latest Publications
- Open Access
- 20 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 2434-2445
A Multi-Stage Structural Captioning Framework for Enhancing Chinese Image-to-Video Generation in Baidu
- 20 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 2434-2445
Open AccessA Multi-Stage Structural Captioning Framework for Enhancing Chinese Image-to-Video Generation in Baidu
- Open Access
- 20 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 2303-2312
Large Vison-Language Foundation Model in Baidu AIGC Image Advertising
- 20 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 2303-2312
Open AccessLarge Vison-Language Foundation Model in Baidu AIGC Image Advertising
- Open Access
- 13 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 4324-4328
Retrieval-Augmented Image Captioning and Generation with Entity Concepts Enhancement for Baidu Multimodal Advertising
- 13 July 2025
- Published by Association for Computing Machinery (ACM)
- p. 4324-4328
Open AccessRetrieval-Augmented Image Captioning and Generation with Entity Concepts Enhancement for Baidu Multimodal Advertising
- Open Access
- 21 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4198-4202
Multi-Stage Refined Visual Captioning for Baidu Ad Creatives Generation
High-quality multimodal training data is of critical importance for improving of multimodal model performance. However, the utilization of web-crawled vision-caption pairs is hindered by the presence of noise and irrelevance, as well as a lack of Chinese data. Large Language Models (LLM) and Large Multimodal Models (LMM) has demonstrated promising performance in cross-modal understanding and generation. In light of this, we propose a Chinese visual captioning pipeline for the synthesis of high-quality data. Our pipeline is comprised of two phases: the initial training of an encoder for visual understanding; and the subsequent fine-tuning of a captioning model in a two-stage iterative human-in-the-loop process, where the captioning model incorporates the pre-trained vision encoder and LLM by a visual cross-attention querying transformer. Extensive experiments have been conducted to validate our framework, including both quantitative and qualitative evaluation of captions generated from images and videos. The synthesis pipeline has been integrated into the ad image creative generation process in Baidu Search Ads, resulting in enhanced capabilities in prompt following.
- 21 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 4198-4202
Open AccessMulti-Stage Refined Visual Captioning for Baidu Ad Creatives Generation
High-quality multimodal training data is of critical importance for improving of multimodal model performance. However, the utilization of web-crawled vision-caption pairs is hindered by the presence of noise and irrelevance, as well as a lack of Chinese data. Large Language Models (LLM) and Large Multimodal Models (LMM) has demonstrated promising performance in cross-modal understanding and generation. In light of this, we propose a Chinese visual captioning pipeline for the synthesis of high-quality data. Our pipeline is comprised of two phases: the initial training of an encoder for visual understanding; and the subsequent fine-tuning of a captioning model in a two-stage iterative human-in-the-loop process, where the captioning model incorporates the pre-trained vision encoder and LLM by a visual cross-attention querying transformer. Extensive experiments have been conducted to validate our framework, including both quantitative and qualitative evaluation of captions generated from images and videos. The synthesis pipeline has been integrated into the ad image creative generation process in Baidu Search Ads, resulting in enhanced capabilities in prompt following.
- Open Access
- 21 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 5102-5110
Scaling Vison-Language Foundation Model to 12 Billion Parameters in Baidu Dynamic Image Advertising
Dynamic image advertising is an add-on service in search advertising that matches visuals to search ads in real-time. However, the image matching system encompasses various sub-tasks with different objectives, increasing the complexity of achieving global optimization. Besides, prevalent long-tailed data poses a challenge to the multimodal representation learning in dynamic image advertising. Recently, vision-language pre-trained models have achieved remarkable performance across a variety of multimodal tasks, and implemented as the foundational representation model in electronic business scenarios. In this paper, to improve multimodal content understanding in Dynamic Image adVERtising, we present a viSion-language rEpresentation model (referred to as DIVERSE) that learns on cross-view and cross-token contrastive loss. Moreover, with large-scale curated advertising image-text data and extensive efficient training techniques, we scale DIVERSE to 12 billion parameters, which is the biggest Chinese multimodal representation model in industrial practices. Experiment results demonstrate the distinct advantages of DIVERSE12B in business datasets, with competitive performance on public benchmarks. Further evaluation in downstream applications including ad text-image retrieval, text-image relevance modeling, and image content moderation, shows that it outperforms previous separately-trained models across offline and online metrics. Moreover, DIVERSE12B has been implemented on the system primary traffic of Baidu Search Ads, bringing considerable increase to both user experience, and revenue for advertisers and search engine.
- 21 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 5102-5110
Open AccessScaling Vison-Language Foundation Model to 12 Billion Parameters in Baidu Dynamic Image Advertising
Dynamic image advertising is an add-on service in search advertising that matches visuals to search ads in real-time. However, the image matching system encompasses various sub-tasks with different objectives, increasing the complexity of achieving global optimization. Besides, prevalent long-tailed data poses a challenge to the multimodal representation learning in dynamic image advertising. Recently, vision-language pre-trained models have achieved remarkable performance across a variety of multimodal tasks, and implemented as the foundational representation model in electronic business scenarios. In this paper, to improve multimodal content understanding in Dynamic Image adVERtising, we present a viSion-language rEpresentation model (referred to as DIVERSE) that learns on cross-view and cross-token contrastive loss. Moreover, with large-scale curated advertising image-text data and extensive efficient training techniques, we scale DIVERSE to 12 billion parameters, which is the biggest Chinese multimodal representation model in industrial practices. Experiment results demonstrate the distinct advantages of DIVERSE12B in business datasets, with competitive performance on public benchmarks. Further evaluation in downstream applications including ad text-image retrieval, text-image relevance modeling, and image content moderation, shows that it outperforms previous separately-trained models across offline and online metrics. Moreover, DIVERSE12B has been implemented on the system primary traffic of Baidu Search Ads, bringing considerable increase to both user experience, and revenue for advertisers and search engine.
- Open Access
- 10 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 2855-2859
Enhancing Baidu Multimodal Advertisement with Chinese Text-to-Image Generation via Bilingual Alignment and Caption Synthesis
Recent advances in generative artificial intelligence have revolutionized information retrieval and content generation, opening up new opportunities for the e-commerce industry. In particular, text-to-image generation models offer a novel approach to guiding the image generation process using natural language input, which is inspiring for multimodal search advertising. Traditional multimodal search ads require advertisers to prepare ad creatives, such as ad images, which is time-consuming and requires uniform image specifications and content quality inspection. To this end, we propose a streamlined generation framework for search ad image creatives. First, we prepare a Chinese image caption model with high-quality image-caption pairs to bootstrap training data refinement. With curated high-quality images and synthesized descriptive captions, we then train a Chinese text-to-image generation model, the largest to date, using SDXL and a 10-billion multimodal text encoder. Specifically, we introduce a two-stage bilingual multimodal representation alignment process to seamlessly integrate the text encoder with the generation model. Extensive experiments validate the effectiveness of our framework, including assessments of image captioning and image generation. The implementation of our framework in Baidu Search Ads shows significant revenue increase, For example, beauty industry ads with generated image creatives achieve a 29% higher click-through rate (CTR).
- 10 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 2855-2859
Open AccessEnhancing Baidu Multimodal Advertisement with Chinese Text-to-Image Generation via Bilingual Alignment and Caption Synthesis
Recent advances in generative artificial intelligence have revolutionized information retrieval and content generation, opening up new opportunities for the e-commerce industry. In particular, text-to-image generation models offer a novel approach to guiding the image generation process using natural language input, which is inspiring for multimodal search advertising. Traditional multimodal search ads require advertisers to prepare ad creatives, such as ad images, which is time-consuming and requires uniform image specifications and content quality inspection. To this end, we propose a streamlined generation framework for search ad image creatives. First, we prepare a Chinese image caption model with high-quality image-caption pairs to bootstrap training data refinement. With curated high-quality images and synthesized descriptive captions, we then train a Chinese text-to-image generation model, the largest to date, using SDXL and a 10-billion multimodal text encoder. Specifically, we introduce a two-stage bilingual multimodal representation alignment process to seamlessly integrate the text encoder with the generation model. Extensive experiments validate the effectiveness of our framework, including assessments of image captioning and image generation. The implementation of our framework in Baidu Search Ads shows significant revenue increase, For example, beauty industry ads with generated image creatives achieve a 29% higher click-through rate (CTR).
- Open Access
- 18 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 3310-3314
Enhancing Dynamic Image Advertising with Vision-Language Pre-training
In the multimedia era, image becomes an effective medium in search advertising. Dynamic Image Advertising (DIA), a system that matches queries with appropriate ad images and generates multimodal ads, is introduced to improve user experience and ad revenue. The core of DIA is a query-image matching module performing ad image retrieval and relevance modeling. Current query-image matching suffers from data scarcity and inconsistency, and insufficient cross-modal fusion. Also, the retrieval and relevance models are separately trained, affecting overall performance. In this paper, we propose a vision-language framework for query-image matching. It consists of two parts. First, we design a base model combining different encoders and tasks, and train it on large-scale image-text pairs to learn general multimodal representation. Then, we fine-tune the base model on advertising business data, unifying relevance modeling and retrieval through multi-objective learning. Our framework has been implemented in Baidu search advertising system "Phoneix Nest". Online evaluation shows that it improves cost per mille (CPM) and click-through rate (CTR) by 1.04% and 1.865% on the system main traffic.
- 18 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 3310-3314
Open AccessEnhancing Dynamic Image Advertising with Vision-Language Pre-training
In the multimedia era, image becomes an effective medium in search advertising. Dynamic Image Advertising (DIA), a system that matches queries with appropriate ad images and generates multimodal ads, is introduced to improve user experience and ad revenue. The core of DIA is a query-image matching module performing ad image retrieval and relevance modeling. Current query-image matching suffers from data scarcity and inconsistency, and insufficient cross-modal fusion. Also, the retrieval and relevance models are separately trained, affecting overall performance. In this paper, we propose a vision-language framework for query-image matching. It consists of two parts. First, we design a base model combining different encoders and tasks, and train it on large-scale image-text pairs to learn general multimodal representation. Then, we fine-tune the base model on advertising business data, unifying relevance modeling and retrieval through multi-objective learning. Our framework has been implemented in Baidu search advertising system "Phoneix Nest". Online evaluation shows that it improves cost per mille (CPM) and click-through rate (CTR) by 1.04% and 1.865% on the system main traffic.
- 3 November 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2568-2575
Predicting user's multi-interests with network embedding in health-related topics
With the rapid growth of Web 2.0, social media has become a prevalent information sharing and seeking channel for health surveillance, in which users form interactive networks by posting and replying messages, providing and rating reviews, attending multiple discussion boards on health-related topics. Users' behaviors in these interactive networks reflect users' multiple interests. To provide better information service for users, it is necessary to analyze the user interactions and predict users' multi-interests. Most existing work in predicting users' multi-interests based on multi label network classification focuses on using approximate inference methods to leverage the dependency information to improve classification results. Inspired by deep learning techniques, DEEPWALK learns label independent latent representations of vertices in a network using local information obtained from truncated random walks, which provides an efficient way for predicting users multi-interests from user interactions. In this paper, we develop a user's multi-interests prediction model based on DEEPWALK, weight information of user interactions is considered when modeling a stream of short constrained random walks and SkipGram is employed to generate more accurate representations of user vertices, which help identify users' interests. Experimental results on two real world health-related datasets show the efficacy of the proposed model.
- 3 November 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2568-2575
Predicting user's multi-interests with network embedding in health-related topics
With the rapid growth of Web 2.0, social media has become a prevalent information sharing and seeking channel for health surveillance, in which users form interactive networks by posting and replying messages, providing and rating reviews, attending multiple discussion boards on health-related topics. Users' behaviors in these interactive networks reflect users' multiple interests. To provide better information service for users, it is necessary to analyze the user interactions and predict users' multi-interests. Most existing work in predicting users' multi-interests based on multi label network classification focuses on using approximate inference methods to leverage the dependency information to improve classification results. Inspired by deep learning techniques, DEEPWALK learns label independent latent representations of vertices in a network using local information obtained from truncated random walks, which provides an efficient way for predicting users multi-interests from user interactions. In this paper, we develop a user's multi-interests prediction model based on DEEPWALK, weight information of user interactions is considered when modeling a stream of short constrained random walks and SkipGram is employed to generate more accurate representations of user vertices, which help identify users' interests. Experimental results on two real world health-related datasets show the efficacy of the proposed model.