Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation

Ранжировщики, судьи и ассистенты: к пониманию взаимодействия больших языковых моделей при оценке информационного поиска
Donald Metzler, Krisztian Balog, Zhen Qin
2025-07-13

AI-generated contentLLM biasLLM judgesLLM-based rankersinformation retrieval evaluation
Large language models (LLMs) are increasingly integral to information retrieval (IR), powering ranking, evaluation, and AI-assisted content creation. This widespread adoption necessitates a critical examination of potential biases arising from the interplay between these LLM-based components. This paper synthesizes existing research and presents novel experiment designs that explore how LLM-based rankers and assistants influence LLM-based judges. We provide the first empirical evidence of LLM judges exhibiting significant bias towards LLM-based rankers. Furthermore, we observe limitations in LLM judges' ability to discern subtle system performance differences. Contrary to some previous findings, our preliminary study does not find evidence of bias against AI-generated content. These results highlight the need for a more holistic view of the LLM-driven information ecosystem. To this end, we offer initial guidelines and a research agenda to ensure the reliable use of LLMs in IR evaluation.
1
A preliminary study finds no evidence that LLM judges are biased against AI-generated content, contrasting with some prior findings.
2
It provides the first empirical evidence that LLM judges exhibit significant bias toward LLM-based rankers.
3
LLM judges show limited ability to distinguish subtle differences in system performance.
4
The paper examines interactions among LLM-based rankers, judges, and assistants within information retrieval evaluation.
5
The paper proposes initial guidelines and a research agenda for reliable LLM-based IR evaluation.

LLM-based information retrieval evaluation ecosystem, comprising rankers, judges, and AI assistants

Interplay-induced biases and performance-discrimination limitations of LLM-based judges, particularly their bias toward LLM-based rankers

Publication Details
Publication Date
2025-07-13
Journal
Publisher
ISSN
Cited by
23
Access Type
Author Information
Authors
Donald Metzler
Krisztian Balog
Zhen Qin
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%