SLSunjae Lee

Sunjae Lee

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Sunjae Lee has a total of 40 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    AgentLens: Adaptive Visual Modalities for Human–Agent Interaction in Mobile GUI Agents

    by Jeonghyeon Kim, Byeongjun Joung, Junwon Lee, Joohyung Lee, Taehoon Min, Sunjae Lee

  • Open Access

    Leveraging LLMs for Efficient and Personalized Smart Home Automation

    by Chaerin Yu, Chihun Choi, Sunjae Lee, Hyosu Kim, Steven Y. Ko, Young-Bae Ko, Sangeun Oh

    The proliferation of smart home devices has increased the complexity of controlling and managing them, leading to user fatigue. In this context, large language models (LLMs) offer a promising solution by enabling natural-language interfaces for Internet of Things (IoT) control. However, existing LLM-based approaches suffer from unreliable and inefficient device control due to the non-deterministic nature of LLMs, high inference latency and cost, and limited personalization. To address these challenges, we present IoTGPT, an LLM-based smart home agent designed to execute IoT commands in a reliable, efficient, and personalized manner. Inspired by how humans manage complex tasks, IoTGPT decomposes user instructions into subtasks and memorizes them. By reusing learned subtasks, subsequent instructions can be processed more efficiently with fewer LLM calls, improving reliability and reducing both latency and cost. IoTGPT also supports fine-grained personalization by adapting individual subtasks to user preferences. Our evaluation demonstrates that IoTGPT outperforms baselines in accuracy, latency/cost, and personalization, while reducing user workload.


  • Open Access

    VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification

    by Jungjae Lee, Dongjae Lee, Chihun Choi, Youngmin Im, Jaeyoung Wi, Kihong Heo, Sangeun Oh, Sunjae Lee, Insik Shin


  • Leveraging Customized Heterogeneous Batteries to Alleviate Low Battery Experience for Mobile Users

    by Jaeheon Kwak, Sunjae Lee, Dae R. Jeong, Arjun Kumar, Dongjae Shin, Ilju Kim, Donghwa Shin, Kilho Lee, Jinkyu Lee, Insik Shin

    Even with advances in single-cell batteries, mobile users still experience low battery anxiety. By analyzing 19,855 hours of user behavior, we propose MixMax, a heterogeneous battery system consisting of three complementary battery types tailored to minimizing low battery time. While the heterogeneous battery system offers an opportunity to simultaneously improve capacity and charging speed, one must face non-trivial challenges to design charge/discharge policies during runtime and determine the ratio of enclosed batteries. They are highly dependent on each other, which entails almost infinite candidates for the choice. MixMax simplifies this by reformulating the problem as an optimization problem, breaking it down into manageable sub-problems. However, MixMax still faces the challenge of catering to all users due to their diverse battery usage patterns. To address this, we introduce a customized MixMax that groups users based on their usage patterns and provides tailored battery solutions. In evaluating MixMax, we fabricate coin-cell batteries, develop a precise battery emulator using the fabricated batteries, and prototype MixMax on a real-world smartphone. Our evaluation shows that MixMax reduces low battery time by up to 24.6% without compromising capacity, volume, weight, or user behavior, and its customized version can further reduce it by up to 46.2%.

  • Open Access

    MobileGPT: Augmenting LLM with Human-like App Memory for Mobile Task Automation

    by Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steve Ko, Sangeun Oh, Insik Shin

    The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT1, an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app---explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task's procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on a dataset of 185 tasks across 18 mobile apps. The results indicate that MobileGPT can automate and learn new tasks with 82.7% accuracy, and is able to adapt them to different contexts with near perfect (98.75%) accuracy while reducing both latency and cost by 62.5% and 68.8%, respectively, compared to the GPT-4 powered baseline.


  • Open Access

    FLUID-IoT : Flexible and Fine-Grained Access Control in Shared IoT Environments via Multi-user UI Distribution

    by Sunjae Lee, Minwoo Jeong, Daye Song, Junyoung Choi, Seoyun Son, Jean Y Song, Insik Shin

    The rapid growth of the Internet of Things (IoT) in shared spaces has led to an increasing demand for sharing IoT devices among multiple users. Yet, existing IoT platforms often fall short by offering an all-or-nothing approach to access control, not only posing security risks but also inhibiting the growth of the shared IoT ecosystem. This paper introduces FLUID-IoT, a framework that enables flexible and granular multi-user access control, even down to the User Interface (UI) component level. Leveraging a multi-user UI distribution technique, FLUID-IoT transforms existing IoT apps into centralized hubs that selectively distribute UI components to users based on their permission levels. Our performance evaluation, encompassing coverage, latency, and memory consumption, affirm that FLUID-IoT can be seamlessly integrated with existing IoT platforms and offers adequate performance for daily IoT scenarios. An in-lab user study further supports that the framework is intuitive and user-friendly, requiring minimal training for efficient utilization.


  • Supporting Flexible and Transparent User Interface Distribution Across Mobile Devices

    by Sangeun Oh, Ahyeon Kim, Sunjae Lee, Kilho Lee, Dae R. Jeong, Steven Y. Ko, Insik Shin

    The growing trend of multi-device ownerships creates opportunities to use applications across devices. However, the current methods of app development/usage remain in the single-device paradigm, which is far below user expectations. For example, it is currently impossible for users to dynamically partition an existing app across different devices to utilize multiple surfaces. We introduce FLUID, a novel multi-device platform that supports simultaneous operation of multiple devices. FLUID aims to i) distribute the user interfaces (UIs) of a single app across multiple devices, ii) support unmodified legacy apps without extra engineering, and iii) support numerous apps with customized UIs. Previous approaches, like screen mirroring and app migration, do not satisfy those goals altogether. However, FLUID is designed to satisfy the goals. It can efficiently deploy UI objects to different devices by identifying only UI states necessary for accurate rendering. And FLUID can execute the distributed UI objects by supporting cross-device method invocations transparently and synchronizing the replicated UIs across devices. Furthermore, FLUID automatically handles unexpected events that may degrade its usability by efficiently maintaining the distributed UIs up to date. Our evaluation using 20 legacy apps shows that FLUID can transparently support numerous apps and is fast enough for interactive use.


  • Open Access

    MixMax: Leveraging Heterogeneous Batteries to Alleviate Low Battery Experience for Mobile Users

    by Jaeheon Kwak, Sunjae Lee, Dae R. Jeong, Arjun Kumar, Dongjae Shin, Ilju Kim, Donghwa Shin, Kilho Lee, Jinkyu Lee, Insik Shin


  • Open Access

    It is Okay to be Distracted: How Real-time Transcriptions Facilitate Online Meeting with Distraction

    by Seoyun Son, Junyoug Choi, Sunjae Lee, Jean Y Song, Insik Shin

    Online meetings are indispensable in collaborative remote work environments, but they are vulnerable to distractions due to their distributed and location-agnostic nature. While distraction often leads to a decrease in online meeting quality due to loss of engagement and context, natural multitasking has positive tradeoff effects, such as increased productivity within a given time unit. In this study, we investigate the impact of real-time transcriptions (i.e., full-transcripts, summaries, and keywords) as a solution to help facilitate online meetings during distracting moments while still preserving multitasking behaviors. Through two rounds of controlled user studies, we qualitatively and quantitatively show that people can better catch up with the meeting flow and feel less interfered with when using real-time transcriptions. The benefits of real-time transcriptions were more pronounced after distracting activities. Furthermore, we reveal additional impacts of real-time transcriptions (e.g., supporting recalling contents) and suggest design implications for future online meeting platforms where these could be adaptively provided to users with different purposes.


  • Towards unobtrusive emotion recognition for affective social communication

    by Hosub Lee, Young Sang Choi, Sunjae Lee, I. P. Park

    Awareness of the emotion of those who communicate with others is a fundamental challenge in building affective intelligent systems. Emotion is a complex state of the mind influenced by external events, physiological changes, or relationships with others. Because emotions can represent a user's internal context or intention, researchers suggested various methods to measure the user's emotions from analysis of physiological signals, facial expressions, or voice. However, existing methods have practical limitations to be used with consumer devices, such as smartphones; they may cause inconvenience to users and require special equipment such as a skin conductance sensor. Our approach is to recognize emotions of the user by inconspicuously collecting and analyzing user-generated data from different types of sensors on the smartphone. To achieve this, we adopted a machine learning approach to gather, analyze and classify device usage patterns, and developed a social network service client for Android smartphones which unobtrusively find various behavioral patterns and the current context of users. Also, we conducted a pilot study to gather real-world data which imply various behaviors and situations of a participant in her/his everyday life. From these data, we extracted 10 features and applied them to build a Bayesian Network classifier for emotion recognition. Experimental results show that our system can classify user emotions into 7 classes such as happiness, surprise, anger, disgust, sadness, fear, and neutral with a surprisingly high accuracy. The proposed system applied to a smartphone demonstrated the feasibility of an unobtrusive emotion recognition approach and a user scenario for emotion-oriented social communication between users.