Jiahao Nick Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jiahao Nick Li has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Jiaju Ma
Stanford University, Stanford, California, USA
2 shared publications - Zhuohao (Jerry) Zhang
Information School, University of Washington, Seattle, Washington, USA
2 shared publications - David Kim
Google, Zurich, Switzerland
1 shared publication - Michelle Li
Reality Labs Research, Meta, Toronto, Ontario, Canada
1 shared publication - Ruofei Du
XR Labs, Google, San Francisco, California, USA
1 shared publication - Stephanie Santosa
Reality Labs Research, Meta, Canada
1 shared publication
Latest Publications
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-20
OmniQuery: Contextually Augmenting Captured Multimodal Memories to Enable Personal Question Answering
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-20
Open AccessOmniQuery: Contextually Augmenting Captured Multimodal Memories to Enable Personal Question Answering
- Open Access
- 13 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-3
OmniQuery: Enabling Question Answering on Personal Memory by Augmenting Multimodal Album Data
We present OmniQuery, an interactive system that augments users’ personal photo albums and enables free-form question answering on users’ past memories. OmniQuery processes multimodal media data in personal albums, aggregates them into related episodic memory databases in different levels, and infers semantic knowledge including personal facts like social relationships, preferences, and experiences. OmniQuery then allows users to interact with their database using natural language, giving media that directly matches the query or an exact answer supported by related media as a result.
- 13 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessOmniQuery: Enabling Question Answering on Personal Memory by Augmenting Multimodal Album Data
We present OmniQuery, an interactive system that augments users’ personal photo albums and enables free-form question answering on users’ past memories. OmniQuery processes multimodal media data in personal albums, aggregates them into related episodic memory databases in different levels, and infers semantic knowledge including personal facts like social relationships, preferences, and experiences. OmniQuery then allows users to interact with their database using natural language, giving media that directly matches the query or an exact answer supported by related media as a result.
- Open Access
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-22
OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-22
Open AccessOmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs
- Open Access
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Human I/O: Towards a Unified Approach to Detecting Situational Impairments
Situationally Induced Impairments and Disabilities (SIIDs) can significantly hinder user experience in contexts such as poor lighting, noise, and multi-tasking. While prior research has introduced algorithms and systems to address these impairments, they predominantly cater to specific tasks or environments and fail to accommodate the diverse and dynamic nature of SIIDs. We introduce Human I/O, a unified approach to detecting a wide range of SIIDs by gauging the availability of human input/output channels. Leveraging egocentric vision, multimodal sensing and reasoning with large language models, Human I/O achieves a 0.22 mean absolute error and a 82% accuracy in availability prediction across 60 in-the-wild egocentric video recordings in 32 different scenarios. Furthermore, while the core focus of our work is on the detection of SIIDs rather than the creation of adaptive user interfaces, we showcase the efficacy of our prototype via a user study with 10 participants. Findings suggest that Human I/O significantly reduces effort and improves user experience in the presence of SIIDs, paving the way for more adaptive and accessible interactive systems in the future.
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Open AccessHuman I/O: Towards a Unified Approach to Detecting Situational Impairments
Situationally Induced Impairments and Disabilities (SIIDs) can significantly hinder user experience in contexts such as poor lighting, noise, and multi-tasking. While prior research has introduced algorithms and systems to address these impairments, they predominantly cater to specific tasks or environments and fail to accommodate the diverse and dynamic nature of SIIDs. We introduce Human I/O, a unified approach to detecting a wide range of SIIDs by gauging the availability of human input/output channels. Leveraging egocentric vision, multimodal sensing and reasoning with large language models, Human I/O achieves a 0.22 mean absolute error and a 82% accuracy in availability prediction across 60 in-the-wild egocentric video recordings in 32 different scenarios. Furthermore, while the core focus of our work is on the detection of SIIDs rather than the creation of adaptive user interfaces, we showcase the efficacy of our prototype via a user study with 10 participants. Findings suggest that Human I/O significantly reduces effort and improves user experience in the presence of SIIDs, paving the way for more adaptive and accessible interactive systems in the future.