Yi Xu

Objectives This study aimed to develop a diagnostic model capable of complementing unacquirable multi-modal ultrasound information to enhance diagnostic performance based on grayscale breast ultrasound (US) alone.Methods Our model was developed and evaluated using a multi-modal internal dataset A, consisting of 2,519 US images, 2,956 ultrasound elastography (USE) images, 957 contrast-enhanced ultrasound (CEUS) videos, and structured text data from 632 breast lesions. We implemented a shared encoder architecture to enhance cross-modal information completion capability, with comparative evaluation against a baseline model lacking shared encoding. External validation was performed on three independently acquired test set B (73 breast lesions)、test set C (62 breast lesions) and D (44 breast lesions) with matched data composition. The diagnostic efficacy of our ModelUT (US+TEXT) was compared with three junior and three senior radiologists in three external test sets. Results In the internal and external test sets, our model successfully enhanced diagnostic performance by compensating for CEUS and/or USE information which were reflected in the modality contribution analysis, exhibited robust multi-modal fusion and completion capabilities compared with the baseline model across all experimental settings (all p < 0.05). For ModelUT, the diagnostic AUCs achieved 0.885 (95% CI: 0.873–0.897) , 0.830 (95% CI:0.813-0.847), 0.856 (95% CI: 0.848–0.863) and 0.813 (95% CI: 0.796–0.827) in the respective test sets, outperforming all radiologists in the three external test sets. Conclusion The proposed information completion model effectively enhanced diagnostic performance of US combined with text data by complementing unacquirable modal information of USE and CEUS.

Text-guided image manipulation has recently attracted significant attention. Prevailing algorithms predominantly focus on modifying the appearances of existing instances, such as texture and attribute editing, while they often fail to address the interactions between different instances or achieve fundamental structural changes, such as multi-object editing. This paper introduces a novel text-guided manipulation task named “relation manipulation”, aimed at fundamentally altering the structure of images. This task is capable of modifying the quantity of instances and, more importantly, enhancing the understanding and editing of interactions among diverse instances. Our approach comprises two main components: relation customization and multi-region guided diffusion. Relation customization fine-tunes specific relationships using a compact dataset of exemplary relations, facilitating nuanced understanding and implementation of instance interactions. Multi-region guided diffusion employs gradient optimization to update the generation process across multiple regions, integrating a fine-grained attention control strategy to minimize regional interference and conflict. Additionally, the demonstrated applications of our method in multi-region inversion underline its potential in practical scenarios, such as relation manipulation of real images and consecutive image manipulation. Compatible with different variants of Stable Diffusion models, our approach seamlessly integrates into the Stable Diffusion WebUI, enabling high-quality image generation and exceptional control over extensive manipulation. This makes it a robust tool for both academic research and creative industries.

Various multi-instance learning (MIL) based approaches have been developed and successfully applied to whole-slide pathological images (WSI). Existing MIL methods emphasize the importance of feature aggregators, but largely neglect the instance-level representation learning. They assume that the availability of a pre-trained feature extractor can be directly utilized or fine-tuned, which is not always the case. This paper proposes to pre-train feature extractor for MIL via a weakly-supervised scheme, i.e., propagating the weak bag-level labels to the corresponding instances for supervised learning. To learn effective features for MIL, we further delve into several key components, including strong data augmentation, a non-linear prediction head and the robust loss function. We conduct experiments on common large-scale WSI datasets and find it achieves better performance than other pre-training schemes (e.g., ImageNet pre-training and self-supervised learning) in different downstream tasks. We further show the compatibility and scalability of the proposed scheme by deploying it in fine-tuning the pathological-specific models and pre-training on merged multiple datasets. To our knowledge, this is the first work focusing on the representation learning for MIL.