Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering
Top Cited Papers
Open Access
- 10 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2905-2909
Abstract
In customer service technical support, swiftly and accurately retrieving relevant past issues is critical for efficiently resolving customer inquiries. The conventional retrieval methods in retrieval-augmented generation (RAG) for large language models (LLMs) treat a large corpus of past issue tracking tickets as plain text, ignoring the crucial intra-issue structure and inter-issue relations, which limits performance. We introduce a novel customer service question-answering method that amalgamates RAG with a knowledge graph (KG). Our method constructs a KG from historical issues for use in retrieval, retaining the intra-issue structure and inter-issue relations. During the question-answering phase, our method parses consumer queries and retrieves related sub-graphs from the KG to generate answers. This integration of a KG not only improves retrieval accuracy by preserving customer service structure information but also enhances answering quality by mitigating the effects of text segmentation. Empirical assessments on our benchmark datasets, utilizing key retrieval (MRR, Recall@K, NDCG@K) and text generation (BLEU, ROUGE, METEOR) metrics, reveal that our method outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU. Our method has been deployed within LinkedIn's customer service team for approximately six months and has reduced the median per-issue resolution time by 28.6%.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Large Language Models on Graphs: A Comprehensive SurveyIEEE Transactions on Knowledge and Data Engineering, 2024
- MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models2024
- Answering Complex Questions by Combining Information from Curated and Extracted Knowledge Bases2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text2018
- Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base2015
- Open Question Answering with Weakly Supervised Embedding Models2014
- Template-based question answering over RDF data2012