On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systems
Authors:
Preprint
Abstract
Large language model multi-agent systems (LLM-MAS) offer a promising paradigm for harnessing collective intelligence to achieve more advanced forms of AI behaviour. While recent studies suggest that LLM-MAS can outperform LLM single-agent systems (LLM-SAS) on certain tasks, the lack of systematic experimental designs limits the strength and generality of these conclusions. We argue that a principled understanding of task complexity, such as the degree of sequential reasoning required and the breadth of capabilities involved, is essential for assessing the effectiveness of LLM-MAS in task solving. To this end, we propose a theoretical framework characterising tasks along two dimensions: depth, representing reasoning length, and width, representing capability diversity. We theoretically examine a representative class of LLM-MAS, namely the multi-agent debate system, and empirically evaluate its performance in both discriminative and generative tasks with varying depth and width. Theoretical and empirical results show that the benefit of LLM-MAS over LLM-SAS increases with both task depth and width, and the effect is more pronounced with respect to depth. This clarifies when LLM-MAS are beneficial and provides a principled foundation for designing future LLM-MAS methods and benchmarks.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- A survey on LLM-based multi-agent systems: workflow, infrastructure, and challengesVicinagearth, 2024
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society2023
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models2023
- Reply to the paper ‘Do not adjust coefficients in Shapley value regression’ by U. Gromping, S. Landau, Applied Stochastic Models in Business and Industry, 2009; DOI: 10.1002/asmb.773Applied Stochastic Models in Business and Industry, 2010