DataSciBench: An LLM Agent Benchmark for Data Science

DataSciBench: бенчмарк LLM-агентов для анализа данных
Wei Wang, Dan Zhang, Jie Tang, Ziniu Hu, Sining Zhoubian, Cai Min, Fengzu Li, Likun Yang, Tianjiao Dong, Yisong Yue
2026-01-01

DataSciBench benchmarkIntention-Function-Code frameworkLLM agentsdata science evaluationsemi-automated ground truth generation
This paper presents DataSciBench, a comprehensive benchmark for evaluating Large Language Models (LLMs) in data science.Unlike existing benchmarks limited to single task, simple evaluation metrics, and readily available ground truth (GT), DataSciBench is built on curated, natural, and challenging prompts with complex evaluation criteria and uncertain GT.To bridge the gap, we develop a semi-automated GT generation pipeline, integrating LLM-based selfconsistency and human verification to ensure accuracy, predefined task types, and aggregate functions (metrics).Furthermore, we introduce an innovative Intention-Function-Code (IFC) framework, assessing code execution outcomes through metrics and programmatic rules.Evaluating 26 models (8 API-based, 8 open-source general, 9 code generation, and 1 agentic models), our approach offers rigorous insights into LLM strengths and weaknesses.Experimental results show API-based models outperform open-source counterparts across all metrics, with DeepAnalyze-8B leading among open-sourced models.We release all code and data at https://github. com/THUDM/DataSciBench/.
1
API-based models outperform open-source models across all metrics, while DeepAnalyze-8B is the strongest open-source model.
2
DataSciBench introduces a comprehensive benchmark for evaluating LLMs on natural, challenging data-science prompts with complex criteria and uncertain ground truth.
3
Evaluation covers 26 models across API-based, open-source general, code-generation, and agentic categories.
4
The Intention-Function-Code framework evaluates code-execution outcomes using task metrics and programmatic rules.
5
The benchmark uses a semi-automated ground-truth pipeline combining LLM self-consistency and human verification to improve evaluation accuracy.

Large Language Models (LLMs) for data science

Their data-science task performance, including code-execution outcomes, evaluated with complex metrics and programmatic criteria

Publication Details
Publication Date
2026-01-01
Journal
Publisher
ISSN
Cited by
85
Access Type
Author Information
Authors
Wei Wang
Dan Zhang
Jie Tang
Ziniu Hu
Sining Zhoubian
Cai Min
Fengzu Li
Likun Yang
Tianjiao Dong
Yisong Yue
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%