FMFan Ma

Fan Ma

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Fan Ma has a total of 43 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing

    by Ruihang Xu, Dewei Zhou, Xiaolong Shen, Fan Ma, Yi Yang

  • Open Access

    CrosSeR: Collaborative Learning for Generalizable Cross-Device Image Super-Resolution via Amplitude Spectrum Sharing

    by Yue Yang, Yunqiu Xu, Anran Li, Saiyu Qi, Fan Ma

    Recent advances in image super-resolution (SR) have enabled high-fidelity reconstruction, yet existing models still underperform on real-world devices due to a distribution gap between public training datasets and personal images. In this study, we investigate a practical yet under-explored scenario, cross-device image SR, where a shared SR model is federatively constructed across multiple devices and endowed with generalization capacity to unseen devices. This new task brings two key challenges: 1) effectively learning from the limited data on individual devices, and 2) developing a generic SR model capable of robustly generalizing to unseen devices. To address these issues, we propose CrosSeR, a novel collaborative learning SR framework. Specifically, an amplitude-mixed curation strategy is first introduced to enrich data diversity on each device. This involves extracting the amplitude and phase spectra of images in the frequency domain, and combining the device’s own phase with amplitude sampled from other devices to enlarge both diversity and scale. The amplitude spectra are globally shared across devices as they do not reveal visual content but contribute to appearance variations. Furthermore, we devise a bi-level joint learning scheme comprising within-device training and cross-device adaptation. This scheme leverages both original and augmented samples to guide the transition from device-specific to device-invariant representation learning. By ensembling the device-specific models, CrosSeR ultimately develops a generic image SR model that effectively generalizes to arbitrary unseen devices. Extensive experiments on two real-world SR benchmarks demonstrate the effectiveness of CrosSeR in achieving both high-quality image reconstruction and strong generalization performance.

  • Open Access

    Empowering Personalized and Privacy-Preserving for Image Super-Resolution with Decentralized Collaboration

    by Yue Yang, Ru Zhang, Fan Ma

    Reconstructing a high-resolution (HR) image from its degraded low-resolution (LR) counterpart facilitates humans to explore more details from the high-quality version. Current image super-resolution (SR) approaches mainly focus on mining richer context through carefully-designed network architectures and optimization objectives within a centralized environment. However, these approaches lack effective data protection and generate unsatisfactory results in practical scenario, due to the degradation gap between public training data (e.g., bicubic downsampling) and real-world personal data captured by different imaging devices. To address these issues, we introduce a novel collaborative learning paradigm, SRGuardian, comprising the personalized SR model for encoding individual-specific imaging information and a shared model for learning general image patterns across individuals. Specifically, SRGuardian inherits a distributed training framework that involves two stages: degradation-aware individual training and personalized parameter fusion. During individual training, we employ a customized degradation generator to encode personalized imaging priors for accurate reconstruction, coupled with a perceptual contrastive regularizer that aligns image representations between the current model and its historical counterparts, thereby enhancing semantic consistency. In the model fusion stage, parameters from personalized models are amalgamated via a hybrid strategy to construct a shared model, where layers responsible for general representation are shared among individuals and layers capturing personal details are retained locally. The integration of these two stages empowers our SRGuardian to achieve both privacy preservation and high-fidelity image rendering. Extensive experiments conducted on two real-world image SR datasets (i.e., RealSR and DRealSR) with various backbones demonstrate that our method significantly surpasses existing approaches in terms of image fidelity and quantitative metrics.

  • VLAB: Enhancing Video Language Pretraining by Feature Adapting and Blending

    by Xingjian He, Sihan Chen, Fan Ma, Zhicheng Huang, Xiaojie Jin, Zikang Liu, Dongmei Fu, Yi Yang, Jing Liu, Jiashi Feng

    Large-scale image-text contrastive pre-training models, such as CLIP, have been demonstrated to effectively learn high-quality multimodal representations. However, there is limited research on learning video-text representations for general video multimodal tasks based on these powerful features. Towards this goal, we propose a novel video-text pre-training method dubbed VLAB: V ideo L anguage pre-training by feature A dapting and B lending, which transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks. Specifically, VLAB is founded on two key strategies: feature adapting and feature blending. In the former, we introduce a new video adapter module to address CLIP's deficiency in modeling temporal information and extend the model's capability to encompass both contrastive and generative tasks. In the latter, we propose an end-to-end training method that further enhances the model's performance by exploiting the complementarity of image and video features. We validate the effectiveness and versatility of VLAB through extensive experiments on highly competitive video multimodal tasks, including video text retrieval, video captioning, and video question answering. Remarkably, VLAB outperforms competing methods significantly and sets new records in video question answering on MSRVTT, MSVD, and TGIF datasets. It achieves an accuracy of 49.6, 60.9, and 79.0, respectively. Our code is available at https://github.com/RainHxj/VLAB .


  • MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis

    by Dewei Zhou, You Li, Fan Ma, Zongxin Yang, Yi Yang

    We introduce the Multi-Instance Generation (MIG) task, which focuses on generating multiple instances within a single image, each accurately placed at predefined positions with attributes such as category, color, and shape, strictly following user specifications. MIG faces three main challenges: avoiding attribute leakage between instances, supporting diverse instance descriptions, and maintaining consistency in iterative generation. To address attribute leakage, we propose the Multi-Instance Generation Controller (MIGC). MIGC generates multiple instances through a divide-and-conquer strategy, breaking down multi-instance shading into single-instance tasks with singular attributes, later integrated. To provide more types of instance descriptions, we developed MIGC++. MIGC++ allows attribute control through text & images and position control through boxes & masks. Lastly, we introduced the Consistent-MIG algorithm to enhance the iterative MIG ability of MIGC and MIGC++. This algorithm ensures consistency in unmodified regions during the addition, deletion, or modification of instances, and preserves the identity of instances when their attributes are changed. We introduce the COCO-MIG and Multimodal-MIG benchmarks to evaluate these methods. Extensive experiments on these benchmarks, along with the COCO-Position benchmark and DrawBench, demonstrate that our methods substantially outperform existing techniques, maintaining precise control over aspects including position, attribute, and quantity. Project page: https://github.com/limuloo/MIGC .


  • Open Access

    VividDreamer: Invariant Score Distillation For Hyper-Realistic Text-to-3D Generation

    by Wenjie Zhuo, Fan Ma, Hehe Fan, Yi Yang

    This paper presents Invariant Score Distillation (ISD), a novel method for high-fidelity text-to-3D generation. ISD aims to tackle the over-saturation and over-smoothing problems in Score Distillation Sampling (SDS). In this paper, SDS is decoupled into a weighted sum of two components: the reconstruction term and the classifier-free guidance term. We experimentally found that over-saturation stems from the large classifier-free guidance scale and over-smoothing comes from the reconstruction term. To overcome these problems, ISD utilizes an invariant score term derived from DDIM sampling to replace the reconstruction term in SDS. This operation allows the utilization of a medium classifier-free guidance scale and mitigates the reconstruction-related errors, thus preventing the over-smoothing and over-saturation of results. Extensive experiments demonstrate that our method greatly enhances SDS and produces realistic 3D objects through single-stage optimization.

  • Top Cited Papers

    MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis

    by Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, Yi Yang

    We present a Multi-Instance Generation (MIG) task, si-multaneously generating multiple instances with diverse controls in one image. Given a set of predefined coordinates and their corresponding descriptions, the task is to ensure that generated instances are accurately at the designated locations and that all instances' attributes adhere to their corresponding description. This broadens the scope of current research on Single-instance generation, elevating it to a more versatile and practical dimension. Inspired by the idea of divide and conquer, we introduce an innovative approach named Multi-Instance Generation Controller (MIGC) to address the challenges of the MIG task. Ini-tially, we break down the MIG task into several subtasks, each involving the shading of a single instance. To ensure precise shading for each instance, we introduce an instance enhancement attention mechanism. Lastly, we aggregate all the shaded instances to provide the necessary information for accurately generating multiple instances in stable diffusion (SD). To evaluate how well generation models per-form on the MIG task, we provide a COCO-MIG bench-mark along with an evaluation pipeline. Extensive experiments were conducted on the proposed COCO-MIG bench-mark, as well as on various commonly used benchmarks. The evaluation results illustrate the exceptional control ca-pabilities of our model in terms of quantity, position, at-tribute, and interaction. Code and demos will be released at https://migcproject.github.io/.


  • Open Access

    FedPAM: Federated Personalized Augmentation Model for Text-to-Image Retrieval

    by Yueying Feng, Fan Ma, Wang Lin, Chang Yao, Jingyuan Chen, Yi Yang

    CLIP-based models have made significant advancements in text-to-image retrieval tasks. However, these retrieval models are typically trained on public datasets with optimizing all parameters, which limits their ability to generalize and adapt quickly to personalized private datasets. In this paper, we introduce a lightweight personalized federated learning solution, namely Federated Personalized Augmentation Model (FedPAM), to achieve personalized text-to-image retrieval from multiple private database. Specifically, for the query text, we fetch the top-k most similar text-image pairs from the private database. We then use an attention-based module to generate personalized representations for different clients. The updated representation includes client-specific information for text-to-image matching, resolving issues of data heterogeneity. Additionally, we ensure efficient and secure communication by fine-tuning a small portion of network parameters. Our experiments demonstrate the effectiveness of the proposed framework, exhibiting a significant performance improvement over recently proposed methods: +5.36 on IAPR TC-12, +2.86 on CC3M, and +1.72 on Flickr30k.