Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Гетерогенные рои: совместная оптимизация ролей и весов моделей для систем на основе нескольких больших языковых моделей
2025-01-01
SCID: 54.1/hr544ada
Discuss with AI
JFK-scoredirected acyclic graphsmodel role optimizationmulti-LLM systemsparticle swarm optimization
Figures from the paper
Abstract (AI)
We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights. We represent multi-LLM systems as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool of LLM experts and a utility function, Heterogeneous Swarms employs two iterative steps: role-step and weight-step. For role-step, we interpret model roles as learning a DAG that specifies the flow of inputs and outputs between LLMs. Starting from a swarm of random continuous adjacency matrices, we decode them into discrete DAGs, call the LLMs in topological order, evaluate on the utility function (e.g. accuracy on a task), and optimize the adjacency matrices with particle swarm optimization based on the utility score. For weight-step, we assess the contribution of individual LLMs in the multi-LLM systems and optimize model weights with swarm intelligence. We propose JFK-score to quantify the individual contribution of each LLM in the best-found DAG of the role-step, then optimize model weights with particle swarm optimization based on the JFK-score. Experiments demonstrate that Heterogeneous Swarms outperforms 15 role- and/or weight-based baselines by 18.5% on average across 12 tasks. Further analysis reveals that Heterogeneous Swarms discovers multi-LLM systems with heterogeneous model roles and substantial collaborative gains, and benefits from the diversity of language models.
Key Findings
1
Across 12 tasks, Heterogeneous Swarms outperforms 15 role- and/or weight-based baselines by 18.5% on average.
2
Heterogeneous Swarms jointly optimizes LLM roles, represented as DAG-based communication structures, and model weights for multi-LLM systems.
3
The discovered systems exhibit heterogeneous model roles, substantial collaborative gains, and improved performance from language-model diversity.
4
The method introduces JFK-score to quantify each LLM’s individual contribution before optimizing model weights through swarm intelligence.
5
The role-optimization step decodes continuous adjacency matrices into discrete DAGs and uses topological message passing with particle swarm optimization.
Research Object
multi-LLM systems represented as directed acyclic graphs of heterogeneous large language models
Research Subject
joint optimization of LLM roles and model weights for collaborative generation and utility-based performance
Publication Details
Publication Date
2025-01-01
Journal
Publisher
ISSN
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest