XJXilin Jiang

Xilin Jiang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Xilin Jiang has a total of 13 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Sci-Phi: A Large Language Model Spatial Audio Descriptor

    by Xilin Jiang, Hannes Gamper, Sebastian Braun

    Acoustic scene perception involves describing the type of sounds, their timing, their direction and distance, as well as their loudness and reverberation. While audio language models excel in sound recognition, single-channel input fundamentally limits spatial understanding. This work presents Sci-Phi, a spatial audio large language model with dual spatial and spectral encoders that estimates a complete parameter set for all sound sources and the surrounding environment. Learning from over 4,000 hours of synthetic first-order Ambisonics recordings including metadata, Sci-Phi enumerates and describes up to four directional sound sources in one pass, alongside non-directional background sounds and room characteristics. We evaluate the model with a permutation-invariant protocol and 15 metrics covering content, location, timing, loudness, and reverberation, and analyze its robustness across source counts, signal-to-noise ratios, reverberation levels, and challenging mixtures of acoustically, spatially, or temporally similar sources. Notably, Sci-Phi generalizes to real room impulse responses with only minor performance degradation. Overall, this work establishes the first audio LLM capable of full spatial-scene description, with strong potential for real-world deployment. Demo: https://sci-phi-audio.github.io/demo


  • Open Access

    Speaker Identity is Robustly Encoded in Spatial Patterns of Intracranial EEG for Attention Decoding

    by Sukru Samet Dindar, Xilin Jiang, Vishal Choudhari, Stephan Bickel, Ashesh Mehta, Catherine Schevon, Guy M. McKhann, Daniel Friedman, Adeen Flinker, Nima Mesgarani

    The human auditory cortex robustly tracks attended speech, yet it remains unclear if speaker identity is encoded in spatial patterns of neural activity independent of temporal dynamics. Here, we demonstrate that the identity of an attended speaker is reliably reflected in distinct, time-invariant spatial activation maps in human intracranial EEG (iEEG). Leveraging these "neural fingerprints", we developed a novel framework for Auditory Attention Decoding (AAD) that shifts from traditional temporal envelope tracking to spatial speaker identification. By decoupling the decoding of "who" is speaking from "when" they are speaking, our modular system achieves state-of-the-art speech extraction, particularly in short time windows (<2 seconds) where temporal models typically fail. Furthermore, we observed a reciprocal shift in neural activity during attentional switches, confirming that these spatial codes dynamically track listener intent. These findings establish that speaker identity is a robust, spatially distributed feature in the auditory cortex, offering a high-speed, complementary mechanism for neuro-steered hearing technologies.


  • Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experience

    by Xilin Jiang, Cong Han, Yinghao Aaron Li, Nima Mesgarani

    In daily life, we encounter a variety of sounds, both desirable and undesirable, with limited control over their presence and volume. Our work introduces “Listen, Chat, and Remix” (LCR), a novel multimodal sound remixer that controls each sound source in a mixture based on user-provided text instructions. LCR distinguishes itself with a user-friendly text interface and its unique ability to remix multiple sound sources simultaneously within a mixture, without needing to separate them. Users input open-vocabulary text prompts, which are interpreted by a large language model to create a semantic filter for remixing the sound mixture. The system then decomposes the mixture into its components, applies the semantic filter, and reassembles filtered components back to the desired output. We developed a 160-hour dataset with over 100k mixtures, including speech and various audio sources, along with text prompts for diverse remixing tasks including extraction, removal, and volume control of single or multiple sources. Our experiments demonstrate significant improvements in signal quality across all remixing tasks and robust performance in zero-shot scenarios with varying numbers and types of sound sources. An audio demo is available at: https://listenchatremix.github.io/demo.