Native Real-Time Interactive Omni Intelligence: A Unified Framework
Preprint
- 19 October 2026
- preprint
- Published by MDPI AG in Preprints.org
Abstract
The rapid progress of large language models has established strong foundations for reasoning, knowledge, and generation, while the integration of vision, audio, video, and other modalities has extended these capabilities toward increasingly unified intelligence across modalities, which we refer to as Omni Intelligence. Yet, real-world assistants must do more than process completed multimodal prompts. They must perceive evolving streams, respond while new evidence is arriving, handle hesitation, overlap, interruption, and changing intent, and coordinate reasoning or tool use that may proceed asynchronously. We refer to this broader capability as \( \textbf{Interactive Omni Intelligence} \): the ability to maintain and exercise omni intelligence under the temporal and adaptive demands of real-time interaction. The question guiding this paper is: How can foundation-model capabilities be carried into real-time interaction while preserving both intelligence and interactivity?We develop a conceptual framework connecting two development paths. One adapts pretrained language or omni models through streaming interfaces, modality modules, post-training, and orchestration. The other develops model-native systems that learn temporal structure, continuous multimodal processing, generation, and interaction control. Across both paths, we compare representations, architectures, fusion, data, training objectives, serving, and evaluation. Their comparison reveals an Intelligence--Interactivity Trade-off between preserving reasoning and knowledge, and meeting the timing, coordination, and resource demands of continuous operation. We examine Multi-Rate Interactive Intelligence as a cross-cutting direction, including foreground--background designs that coordinate fast interaction with slower reasoning, retrieval, memory, and tools. Finally, we identify open problems in evolving state, persistent and proactive interaction, joint evaluation, and safe open-world operation. The resulting framework connects capability preservation, interaction control, and multi-rate coordination.
Keywords
All Related Versions
This publication has 210 references indexed in Scilit:
- Dialospeech: Dual-Speaker Dialogue Generation with LLM and Flow Matching2025
- Tool learning with large language models: a surveyFrontiers of Computer Science, 2025
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- A survey on multimodal large language modelsNational Science Review, 2024
- LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models2024
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- Turn-taking in Conversational Systems and Human-Robot Interaction: A ReviewComputer Speech & Language, 2021
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding2016
- Turn-taking in Human Communication – Origins and Implications for Language ProcessingTrends in Cognitive Sciences, 2016
- Universals and cultural variation in turn-taking in conversationProceedings of the National Academy of Sciences, 2009