PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figures
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (19) , 20488-20496
Abstract
Writing comprehensive and accurate descriptions of technical drawings in patent documents is crucial to effective knowledge sharing and enabling the replication and protection of intellectual property. However, automation of this task has been largely overlooked by the research community. To this end, we introduce PatentDesc-355K, a novel large-scale dataset containing ∼355K patent figures along with their brief and detailed textual descriptions extracted from more than 60K US patent documents. In addition, we propose PatentLMM – a novel large multimodal model specifically tailored to generate high-quality descriptions of patent figures. Our proposed PatentLMM comprises two key components: (i) PatentMME, a specialized multimodal vision encoder that captures the unique structural elements of patent figures, and (ii) PatentLLaMA, a domain-adapted version of LLaMA fine-tuned on a large collection of patents. Our extensive experiments demonstrate that training a vision encoder specifically designed for patent figures significantly boosts the performance, generating coherent descriptions compared to fine-tuning similar-sized off-the-shelf multimodal models. PatentDesc-355K and PatentLMM pave the way for automating the understanding of patent figures, enabling efficient knowledge sharing and faster drafting of patent documents.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Improved Baselines with Visual Instruction Tuning2024
- LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding2024
- Towards Making Flowchart Images Machine Interpretable2023
- Taming Transformers for High-Resolution Image Synthesis2021
- TextCaps: A Dataset for Image Captioning with Reading Comprehension2020
- nocaps: novel object captioning at scale2019
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning2017
- Show and tell: A neural image caption generator2015
- Microsoft COCO: Common Objects in Context2014