AutoJournaling: A Context-Aware Journaling System Leveraging MLLMs on Smartphone Screenshots
Open Access
- 4 December 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2347-2352
Abstract
Journaling offers significant benefits, including fostering self-reflection, enhancing writing skills, and aiding in mood monitoring. However, many people abandon the practice because traditional journaling is time-consuming, and detailed life events may be overlooked if not recorded promptly. Given that smartphones are the most widely used devices for entertainment, work, and socialization, they present an ideal platform for innovative approaches to journaling. Despite their ubiquity, the potential of using digital phenotyping, a method of unobtrusively collecting data from digital devices to gain insights into psychological and behavioral patterns, for automated journal generation has been largely underexplored. In this study, we propose AutoJournaling, the first-of-its-kind system that automatically generates journals by collecting and analyzing screenshots from smartphones. This system captures life events and corresponding emotions, offering a novel approach to digital phenotyping. We evaluated AutoJournaling by collecting screenshots every 3 seconds from three students over five days, demonstrating its feasibility and accuracy. AutoJournaling is the first framework to utilize seamlessly collected screenshots for journal generation, providing new insights into psychological states through digital phenotyping.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Contextual AI Journaling: Integrating LLM and Time Series Behavioral Sensing Technology to Promote Self-Reflection and Well-being using the MindScape App2024
- MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling2024
- DiaryMate: Understanding User Perceptions and Experience in Human-AI Collaboration for Personal Journaling2024
- Federated Few-Shot Learning for Mobile NLP2023
- AWARE-Light: a smartphone tool for experience sampling and digital phenotypingPersonal and Ubiquitous Computing, 2022
- Cosmo2022
- Screenomics: A Framework to Capture and Analyze Personal Life Experiences and the Ways that Technology Shapes ThemHuman–Computer Interaction, 2019
- Examining Unlock Journaling with Diaries and Reminders for In Situ Self-Report in Health and Wellness2016