Evolving Multi-Agent Systems with Large Language Models: Surrogate Fitness Limitations and Empirical Insights
Authors:
- Vadim AkhmerovVAVadim Akhmerov
- Dmitry GilemkhanovDGDmitry Gilemkhanov
- Jerzy KamińskiJerzy Kamiński
- Ivan DubrovskyIDIvan Dubrovsky
- Maria ZaitsevaMZMaria Zaitseva
- Dmitrii KuznechenkoDKDmitrii Kuznechenko
- Egor RozhdestvenskiyEREgor Rozhdestvenskiy
- Danil FedorovDFDanil Fedorov
- Ilia RevinIRIlia Revin
- Vadim PotemkinVPVadim Potemkin
- Ravil MussabayevRMRavil Mussabayev
- Alexander BoukhanovskyABAlexander Boukhanovsky
Open Access
- 13 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1344-1352
Abstract
We investigate the use of large language models (LLMs) as evolutionary operators for optimizing the architecture of multi-agent systems (MAS) - a representative instance of LLM-driven optimization over attributed-graph artifacts that arise in generative design. Given a task description, an initial MAS configuration is generated and then refined on a per-task basis through two optimization strategies: (1) a population-based evolutionary approach where LLMs serve as mutation, crossover, and selection operators, with an LLM-as-a-Judge ensemble providing the surrogate fitness signal; and (2) a simpler iterative regeneration scheme that uses textual feedback from LLM judges to guide single-solution improvement without maintaining a population. We evaluate both approaches on the GAIA benchmark. Our experiments reveal a critical sensitivity to the judge model's quality: a capable judge (Claude 4 Sonnet) yields +14.3 and +6.67 percentage-point accuracy improvements for evolutionary and iterative approaches, respectively, whereas a weaker judge (Gemini-2.5-Flash) provides no benefit or even degrades performance below the unoptimized baseline. Both approaches incur substantial computational costs, raising questions about practical cost-effectiveness. These results highlight fundamental challenges of surrogate fitness functions in LLM-driven evolutionary optimization and offer practical insights for applying evolutionary methods to MAS architecture design.Keywords
Funding Information
- Russian Science Foundation (25-11-00367)
This publication has 4 references indexed in Scilit:
- Mathematical discoveries from program search with large language modelsNature, 2023
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena2023
- Machine learning enables design automation of microfluidic flow-focusing droplet generationNature Communications, 2021
- Surrogate-assisted evolutionary computation: Recent advances and future challengesSwarm and Evolutionary Computation, 2011