EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association
Authors:
- Chen LuoCLChen Luo
- Sheikh Muhammad SarwarSSSheikh Muhammad Sarwar
Preprint
Abstract
Goal-oriented script planning, or the ability to devise coherent sequences of actions toward specific goals, is commonly employed by humans to plan for typical activities. In e-commerce, customers increasingly seek LLM-based assistants to generate scripts and recommend products at each step, thereby facilitating convenient and efficient shopping experiences. However, this capability remains underexplored due to several challenges, including the inability of LLMs to simultaneously conduct script planning and product retrieval, difficulties in matching products caused by semantic discrepancies between planned actions and search queries, and a lack of methods and benchmark data for evaluation. In this paper, we step forward by formally defining the task of E-commerce Script Planning (EcomScript) as three sequential subtasks. We propose a novel framework that enables the scalable generation of product-enriched scripts by associating products with each step based on the semantic similarity between the actions and their purchase intentions. By applying our framework to real-world e-commerce data, we construct the very first large-scale EcomScript dataset, EcomScriptBench, which includes 605,229 scripts sourced from 2.4 million products. Human annotations are then conducted to provide gold labels for a sampled subset, forming an evaluation benchmark. Extensive experiments reveal that current (L)LMs face significant challenges with EcomScript tasks, even after fine-tuning, while injecting product purchase intentions improves their performance.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Miko: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery2024
- COSMO: A Large-Scale E-commerce Common Sense Knowledge Generation and Serving System at Amazon2024
- FLTRNN: Faithful Long-Horizon Task Planning for Robotics with Large Language Models2024
- Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models2024
- The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI2022
- Dy-HIEN2022
- DISCOS: Bridging the Gap between Discourse Knowledge and Commonsense Knowledge2021
- Mining User Intentions from Medical Queries2016
- Detecting online commercial intention (OCI)2006
- Price, Product Information, and Purchase Intention: An Empirical StudyJournal of the Academy of Marketing Science, 1994