Mingwen Wang

Visual storytelling (VIST) involves generating coherent, creative, and vivid narrative for a collection of images. It remains an immense challenge within cross-modal domain. Traditional mainstream storytelling work were less proficient in handling long sequential relationships. Though large-scale visual-language pre-training (VLP) models demonstrated promising prospect on cross-modal tasks. So far they still were not particularly adept at handling tasks involving image sequences. Moreover, the reference descriptions in the available VIST benchmark dataset are short and simplistic, which constrains model’s potential capabilities. Current models struggle to produce truly rich and vivid narratives. Therefore, in this article, we will address these deficiencies, and contribute from both dataset and innovative model aspects. Firstly, by leveraging large language model (LLM), we have constructed a new dataset VIST++ which can enrich vivid narratives for open-ended image sequences. The dataset has potential on providing beneficial support for future model learning. Secondly, we have proposed an innovative auto-regressive story generation model named ReStoryGen. It can be applied to image sequences of varying lengths in an open-ended way. We have performed extensive experiments and evaluations in terms of visual grounding, coherence and non-redundancy. The experiment results have convincingly demonstrated ReStoryGen achieves impressive outcomes through utilizing parameter-efficient instruction-tuning. Related source codes and models are distributed on Github https://github.com/lixinliu1995/story_gen .

Clothing image synthesis has emerged as a crucial technology in the fashion domain, enabling designers to rapidly transform creative concepts into realistic visual representations. However, the existing methods struggle to effectively integrate multiple guiding information sources, such as sketches and texture patches, limiting their ability to precisely control the generated content. This often results in issues such as semantic inconsistencies and the loss of fine-grained texture details, which significantly hinders the advancement of this technology. To address these issues, we propose the Fast Fourier Asymmetric Context Aggregation Network (FCAN), a novel image generation network designed to achieve controllable clothing image synthesis guided by design sketches and texture patches. In the FCAN, we introduce the Asymmetric Context Aggregation Mechanism (ACAM), which leverages multi-scale and multi-stage heterogeneous features to achieve efficient global visual context modeling, significantly enhancing the model’s ability to integrate guiding information. Complementing this, the FCAN also incorporates a Fast Fourier Channel Dual Residual Block (FF-CDRB), which utilizes the frequency-domain properties of Fast Fourier Convolution to enhance fine-grained content inference while maintaining computational efficiency. We evaluate the FCAN on the newly constructed SKFashion dataset and the publicly available VITON-HD and Fashion-Gen datasets. The experimental results demonstrate that the FCAN consistently generates high-quality clothing images aligned with the design intentions while outperforming the baseline methods across multiple performance metrics. Furthermore, the FCAN demonstrates superior robustness to varying texture conditions compared to the existing methods, highlighting its adaptability to diverse real-world scenarios. These findings underscore the potential of the FCAN to advance this technology by enabling controllable and high-quality image generation.