Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?
Preprint
Abstract
The capabilities of pretrained language models have opened opportunities to explore new application areas, but applications involving human-human interaction are limited by the fact that most data is protected from public release for privacy reasons. Problem-solving human dialogues in real applications can be much more complex than existing Wizard-of-Oz collections, preventing successful domain transfer. To support information extraction (IE) for a private call center dataset, we introduce a human-in-the-loop dialogue generation framework capable of synthesizing realistic dialogues. In IE experiments with auto insurance call center dialogues, we observe 25\% relative improvement in $F_1$ after augmenting a small set of real human conversations with synthetic data. We release code and our synthetic dataset to illustrate the complexity of real-world call center conversations and encourage development of complex dialogue datasets that are more representative of natural data.
Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Generate, Annotate, and Learn: NLP with Synthetic TextTransactions of the Association for Computational Linguistics, 2022
- Privacy-Preserving Synthetic Data Generation for Recommendation Systems2022
- SinGAN-Seg: Synthetic training data generation for medical image segmentationPLOS ONE, 2022
- Differentially private synthetic medical data generation using convolutional GANsInformation Sciences, 2022
- Annotation Inconsistency and Entity Bias in MultiWOZ2021
- Dialogue State Tracking with a Language Model using Schema-Driven Prompting2021
- All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text2021
- Private FL-GAN: Differential Privacy Synthetic Data Generation Based on Federated Learning2020
- Data synthesis based on generative adversarial networksProceedings of the VLDB Endowment, 2018
- An iterative design methodology for user-friendly natural language office information applicationsACM Transactions on Information Systems, 1984