Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems

Гетерогенные рои: совместная оптимизация ролей и весов моделей для систем на основе нескольких больших языковых моделей
Shangbin Feng, Zifeng Wang, Palash Goyal, Yike Wang, Weijia Shi, Huang Xia, Hamid Palangi, Luke Zettlemoyer, Yulia Tsvetkov, Chen‐Yu Lee, Tomas Pfister
2025-01-01

JFK-scoredirected acyclic graphsmodel role optimizationmulti-LLM systemsparticle swarm optimization
We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights. We represent multi-LLM systems as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool of LLM experts and a utility function, Heterogeneous Swarms employs two iterative steps: role-step and weight-step. For role-step, we interpret model roles as learning a DAG that specifies the flow of inputs and outputs between LLMs. Starting from a swarm of random continuous adjacency matrices, we decode them into discrete DAGs, call the LLMs in topological order, evaluate on the utility function (e.g. accuracy on a task), and optimize the adjacency matrices with particle swarm optimization based on the utility score. For weight-step, we assess the contribution of individual LLMs in the multi-LLM systems and optimize model weights with swarm intelligence. We propose JFK-score to quantify the individual contribution of each LLM in the best-found DAG of the role-step, then optimize model weights with particle swarm optimization based on the JFK-score. Experiments demonstrate that Heterogeneous Swarms outperforms 15 role- and/or weight-based baselines by 18.5% on average across 12 tasks. Further analysis reveals that Heterogeneous Swarms discovers multi-LLM systems with heterogeneous model roles and substantial collaborative gains, and benefits from the diversity of language models.
1
Across 12 tasks, Heterogeneous Swarms outperforms 15 role- and/or weight-based baselines by 18.5% on average.
2
Heterogeneous Swarms jointly optimizes LLM roles, represented as DAG-based communication structures, and model weights for multi-LLM systems.
3
The discovered systems exhibit heterogeneous model roles, substantial collaborative gains, and improved performance from language-model diversity.
4
The method introduces JFK-score to quantify each LLM’s individual contribution before optimizing model weights through swarm intelligence.
5
The role-optimization step decodes continuous adjacency matrices into discrete DAGs and uses topological message passing with particle swarm optimization.

multi-LLM systems represented as directed acyclic graphs of heterogeneous large language models

joint optimization of LLM roles and model weights for collaborative generation and utility-based performance

Publication Details
Publication Date
2025-01-01
Journal
Publisher
ISSN
Access Type
Author Information
Authors
Shangbin Feng
Zifeng Wang
Palash Goyal
Yike Wang
Weijia Shi
Huang Xia
Hamid Palangi
Luke Zettlemoyer
Yulia Tsvetkov
Chen‐Yu Lee
Tomas Pfister
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%