Dong Wang

While intelligent virtual assistants like Siri, Alexa, and Google Assistant have become ubiquitous in modern life, they still face limitations in their ability to follow multi-step instructions and accomplish complex goals articulated in natural language. However, recent breakthroughs in large language models (LLMs) show promise for overcoming existing barriers by enhancing natural language processing and reasoning capabilities. Though promising, applying LLMs to create more advanced virtual assistants still faces challenges like ensuring robust performance and handling variability in real-world user commands. This paper proposes a novel LLM-based virtual assistant that can automatically perform multi-step operations within mobile apps based on high-level user requests. The system represents an advance in assistants by providing an end-to-end solution for parsing instructions, reasoning about goals, and executing actions. LLM-based Process Automation (LLMPA) has modules for decomposing instructions, generating descriptions, detecting interface elements, predicting next actions, and error checking. Experiments demonstrate the system completing complex mobile operation tasks in Alipay based on natural language instructions. This showcases how large language models can enable automated assistants to accomplish real-world tasks. The main contributions are the novel LLMPA architecture optimized for app process automation, the methodology for applying LLMs to mobile apps, and demonstrations of multi-step task completion in a real-world environment. Notably, this work represents the first real-world deployment and extensive evaluation of a large language model-based virtual assistant in a widely used mobile application with an enormous user base numbering in the hundreds of millions.

When a customer sees a movie recommendation, she may buy the ticket right away, which is the immediate feedback that helps improve the recommender system. Alternatively, she may choose to come back later and this long-term feedback is also modeled to promote user retention. However, the long-term feedback comes with non-trivial challenges in understanding user retention: the complicated correlation between current demands and follow-up demands, coupled with the periodicity of services. For instance, before the movie, the customer buys popcorn through the App, which temporally correlates with the initial movie recommendation. Days later, she checks the App for new movies, as a weekly routine. To address this complexity in a more fine-grained revisit modeling, we propose Time Aware Service Sequential Recommendation (TASSR) for user retention, which is equipped with a multi-task design and an In-category TimeSeqBlock module. Large-scale online and offline experiments demonstrate its significant advantages over competitive baselines.