Are Human Conversations Special? A Large Language Model Perspective
Preprint
Abstract
This study analyzes changes in the attention mechanisms of large language models (LLMs) when used to understand natural conversations between humans (human-human). We analyze three use cases of LLMs: interactions over web content, code, and mathematical texts. By analyzing attention distance, dispersion, and interdependency across these domains, we highlight the unique challenges posed by conversational data. Notably, conversations require nuanced handling of long-term contextual relationships and exhibit higher complexity through their attention patterns. Our findings reveal that while language models exhibit domain-specific attention behaviors, there is a significant gap in their ability to specialize in human conversations. Through detailed attention entropy analysis and t-SNE visualizations, we demonstrate the need for models trained with a diverse array of high-quality conversational data to enhance understanding and generation of human-like dialogue. This research highlights the importance of domain specialization in language models and suggests pathways for future advancement in modeling human conversational nuances.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains2024
- Enhanced Speaker-Aware Multi-Party Multi-Turn Dialogue ComprehensionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- AIR-Act2Act: Human–human interaction dataset for teaching non-verbal social behaviors to robotsThe International Journal of Robotics Research, 2021
- Dialogue State Tracking with a Language Model using Schema-Driven Prompting2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Analyzing the Structure of Attention in a Transformer Language Model2019
- Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems2019
- Global-Locally Self-Attentive Encoder for Dialogue State Tracking2018
- An Analysis of Encoder Representations in Transformer-Based Machine Translation2018
- Authentic Language and Authentic Conversational TextsForeign Language Annals, 1986