Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation
Ранжировщики, судьи и ассистенты: к пониманию взаимодействия больших языковых моделей при оценке информационного поиска
2025-07-13
SCID: 54.1/b3pnyak6
Discuss with AI
AI-generated contentLLM biasLLM judgesLLM-based rankersinformation retrieval evaluation
Figures from the paper
Abstract (AI)
Large language models (LLMs) are increasingly integral to information retrieval (IR), powering ranking, evaluation, and AI-assisted content creation. This widespread adoption necessitates a critical examination of potential biases arising from the interplay between these LLM-based components. This paper synthesizes existing research and presents novel experiment designs that explore how LLM-based rankers and assistants influence LLM-based judges. We provide the first empirical evidence of LLM judges exhibiting significant bias towards LLM-based rankers. Furthermore, we observe limitations in LLM judges' ability to discern subtle system performance differences. Contrary to some previous findings, our preliminary study does not find evidence of bias against AI-generated content. These results highlight the need for a more holistic view of the LLM-driven information ecosystem. To this end, we offer initial guidelines and a research agenda to ensure the reliable use of LLMs in IR evaluation.
Key Findings
1
A preliminary study finds no evidence that LLM judges are biased against AI-generated content, contrasting with some prior findings.
2
It provides the first empirical evidence that LLM judges exhibit significant bias toward LLM-based rankers.
3
LLM judges show limited ability to distinguish subtle differences in system performance.
4
The paper examines interactions among LLM-based rankers, judges, and assistants within information retrieval evaluation.
5
The paper proposes initial guidelines and a research agenda for reliable LLM-based IR evaluation.
Research Object
LLM-based information retrieval evaluation ecosystem, comprising rankers, judges, and AI assistants
Research Subject
Interplay-induced biases and performance-discrimination limitations of LLM-based judges, particularly their bias toward LLM-based rankers
Publication Details
Publication Date
2025-07-13
Journal
Publisher
ISSN
Cited by
23
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest