Tuochao Chen
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Tuochao Chen has a total of 25 co-authors. Here are the top co-authors by number of shared publications:
- Shyamnath Gollakota
University of Washington, Seattle, USA
10 shared publications - Malek Itani
University of Washington, Seattle, USA
5 shared publications Shyamnath Gollakota
School of Computer Science & Engineering University of Washington
5 shared publications- Justin Chan
Carnegie Mellon University, Pittsburgh, USA
4 shared publications Malek Itani
School of Computer Science & Engineering University of Washington
3 shared publications- Takuya Yoshioka
Microsoft Redmond WA USA
3 shared publications
Latest Publications
- Open Access
Proactive Hearing Assistants that Isolate Egocentric Conversations
We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: https://proactivehearing.cs.washington.edu/
Open AccessProactive Hearing Assistants that Isolate Egocentric Conversations
We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: https://proactivehearing.cs.washington.edu/
- Open Access
AV-Dialog: Spoken Dialogue Models with Audio-Visual Input
Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.
Open AccessAV-Dialog: Spoken Dialogue Models with Audio-Visual Input
Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.
- Open Access
- 3 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 863-877
Wireless Hearables With Programmable Speech AI Accelerators
- 3 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 863-877
Open AccessWireless Hearables With Programmable Speech AI Accelerators
- Open Access
TF-MLPNet: Tiny Real-Time Neural Speech Separation
Speech separation on hearable devices can enable transformative augmented and enhanced hearing capabilities. However, state-of-the-art speech separation networks cannot run in real-time on tiny, low-power neural accelerators designed for hearables, due to their limited compute capabilities. We present TF-MLPNet, the first speech separation network capable of running in real-time on such low-power accelerators while outperforming existing streaming models for blind speech separation and target speech extraction. Our network operates in the time-frequency domain, processing frequency sequences with stacks of fully connected layers that alternate along the channel and frequency dimensions, and independently processing the time sequence at each frequency bin using convolutional layers. Results show that our mixed-precision quantization-aware trained (QAT) model can process 6 ms audio chunks in real-time on the GAP9 processor, achieving a 3.5-4x runtime reduction compared to prior speech separation models.
Open AccessTF-MLPNet: Tiny Real-Time Neural Speech Separation
Speech separation on hearable devices can enable transformative augmented and enhanced hearing capabilities. However, state-of-the-art speech separation networks cannot run in real-time on tiny, low-power neural accelerators designed for hearables, due to their limited compute capabilities. We present TF-MLPNet, the first speech separation network capable of running in real-time on such low-power accelerators while outperforming existing streaming models for blind speech separation and target speech extraction. Our network operates in the time-frequency domain, processing frequency sequences with stacks of fully connected layers that alternate along the channel and frequency dimensions, and independently processing the time sequence at each frequency bin using convolutional layers. Results show that our mixed-precision quantization-aware trained (QAT) model can process 6 ms audio chunks in real-time on the GAP9 processor, achieving a 3.5-4x runtime reduction compared to prior speech separation models.
- Open Access
SoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extraction
This paper introduces SoundSculpt, a neural network designed to extract target sound fields from ambisonic recordings. SoundSculpt employs an ambisonic-in-ambisonic-out architecture and is conditioned on both spatial information (e.g., target direction obtained by pointing at an immersive video) and semantic embeddings (e.g., derived from image segmentation and captioning). Trained and evaluated on synthetic and real ambisonic mixtures, SoundSculpt demonstrates superior performance compared to various signal processing baselines. Our results further reveal that while spatial conditioning alone can be effective, the combination of spatial and semantic information is beneficial in scenarios where there are secondary sound sources spatially close to the target. Additionally, we compare two different semantic embeddings derived from a text description of the target sound using text encoders.
Open AccessSoundSculpt: Direction and Semantics Driven Ambisonic Target Sound Extraction
This paper introduces SoundSculpt, a neural network designed to extract target sound fields from ambisonic recordings. SoundSculpt employs an ambisonic-in-ambisonic-out architecture and is conditioned on both spatial information (e.g., target direction obtained by pointing at an immersive video) and semantic embeddings (e.g., derived from image segmentation and captioning). Trained and evaluated on synthetic and real ambisonic mixtures, SoundSculpt demonstrates superior performance compared to various signal processing baselines. Our results further reveal that while spatial conditioning alone can be effective, the combination of spatial and semantic information is beneficial in scenarios where there are secondary sound sources spatially close to the target. Additionally, we compare two different semantic embeddings derived from a text description of the target sound using text encoders.
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-19
Spatial Speech Translation: Translating Across Space With Binaural Hearables
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-19
Open AccessSpatial Speech Translation: Translating Across Space With Binaural Hearables
- 31 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Pervasive Computing
- Vol. 23 (4) , 53-55
Creating Super-Hearing Capabilities With Real-Time AI
Sound is a fundamental medium through which we perceive our environment. However, today we are surrounded by a cacophony of sounds that can overwhelm our senses. Indeed, human auditory perception can be limited in noisy environments. Imagine being in a crowded room with a cacophony of sounds and having the ability to focus on or remove sounds from a specific person or based on their semantic descriptions. This requires understanding and manipulating an acoustic scene, isolating each sound, and associating a spatial context or semantic meaning with each constituent sound or speaker—a challenging set of tasks even for the human brain in noisy environments.
- 31 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Pervasive Computing
- Vol. 23 (4) , 53-55
Creating Super-Hearing Capabilities With Real-Time AI
Sound is a fundamental medium through which we perceive our environment. However, today we are surrounded by a cacophony of sounds that can overwhelm our senses. Indeed, human auditory perception can be limited in noisy environments. Imagine being in a crowded room with a cacophony of sounds and having the ability to focus on or remove sounds from a specific person or based on their semantic descriptions. This requires understanding and manipulating an acoustic scene, isolating each sound, and associating a spatial context or semantic meaning with each constituent sound or speaker—a challenging set of tasks even for the human brain in noisy environments.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 25377-25394
Proactive Hearing Assistants that Isolate Egocentric Conversations
Guilin Hu, Malek Itani, Tuochao Chen, Shyamnath Gollakota. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 25377-25394
Open AccessProactive Hearing Assistants that Isolate Egocentric Conversations
Guilin Hu, Malek Itani, Tuochao Chen, Shyamnath Gollakota. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 13801-13824
LlamaPIE: Proactive In-Ear Conversation Assistants
Tuochao Chen, Nicholas Scott Batchelder, Alisa Liu, Noah A. Smith, Shyamnath Gollakota. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 13801-13824
Open AccessLlamaPIE: Proactive In-Ear Conversation Assistants
Tuochao Chen, Nicholas Scott Batchelder, Alisa Liu, Noah A. Smith, Shyamnath Gollakota. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- 14 November 2024
- Published by Springer Nature in Nature Electronics
- Vol. 7 (11) , 1047-1058
Hearable devices with sound bubbles
- 14 November 2024
- Published by Springer Nature in Nature Electronics
- Vol. 7 (11) , 1047-1058
Hearable devices with sound bubbles