SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

SafeScientist: к выявлению научных открытий с учетом рисков при использовании агентов LLM
Jiaxuan You, Kunlun Zhu, Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Shang, Nuoxing, Zijia Liu, Han, Peixuan, Yue Su, Haofei Yu, Jiaxuan You, Zijia Liu, Haofei Yu
2025-05-29

AI scientist frameworkLLM agentsSafeScientistSciSafetyBenchadversarial attack robustnessagent-collaboration monitoringethical reviewerhigh-risk scientific tasksprompt monitoringsafety pipelinetool-related risk taskstool-use monitoring
Recent advancements in large language model (LLM) agents have significantly accelerated scientific discovery automation, yet concurrently raised critical ethical and safety concerns. To systematically address these challenges, we introduce \textbf{SafeScientist}, an innovative AI scientist framework explicitly designed to enhance safety and ethical responsibility in AI-driven scientific exploration. SafeScientist proactively refuses ethically inappropriate or high-risk tasks and rigorously emphasizes safety throughout the research process. To achieve comprehensive safety oversight, we integrate multiple defensive mechanisms, including prompt monitoring, agent-collaboration monitoring, tool-use monitoring, and an ethical reviewer component. Complementing SafeScientist, we propose \textbf{SciSafetyBench}, a novel benchmark specifically designed to evaluate AI safety in scientific contexts, comprising 240 high-risk scientific tasks across 6 domains, alongside 30 specially designed scientific tools and 120 tool-related risk tasks. Extensive experiments demonstrate that SafeScientist significantly improves safety performance by 35\% compared to traditional AI scientist frameworks, without compromising scientific output quality. Additionally, we rigorously validate the robustness of our safety pipeline against diverse adversarial attack methods, further confirming the effectiveness of our integrated approach. The code and data will be available at https://github.com/ulab-uiuc/SafeScientist. \textcolor{red}{Warning: this paper contains example data that may be offensive or harmful.}
1
SafeScientist improves safety performance by 35% compared to traditional AI scientist frameworks while maintaining scientific output quality.
2
SafeScientist integrates multiple defensive mechanisms: prompt monitoring, agent-collaboration monitoring, tool-use monitoring, and an ethical reviewer component.
3
SafeScientist is an AI scientist framework that proactively refuses ethically inappropriate or high-risk scientific tasks.
4
SciSafetyBench is a new benchmark with 240 high-risk scientific tasks across 6 domains, 30 scientific tools, and 120 tool-related risk tasks for evaluating AI safety in scientific contexts.
5
The SafeScientist safety pipeline is robust against diverse adversarial attack methods, validating the effectiveness of the integrated approach.

SafeScientist AI scientist framework for risk-aware scientific discovery (including its defensive components: prompt monitoring, agent-collaboration monitoring, tool-use monitoring, and ethical reviewer)

Improving safety and ethical responsibility in AI-driven scientific exploration by detecting/refusing high-risk or ethically inappropriate tasks and integrating multi-layer defensive mechanisms, evaluated via the SciSafetyBench benchmark and robustness tests against adversarial attacks

Publication Details
Publication Date
2025-05-29
Journal
Publisher
ISSN
Cited by
2
Access Type
Author Information
Authors
Jiaxuan You
Kunlun Zhu
Kunlun Zhu
Jiaxun Zhang
Ziheng Qi
Shang, Nuoxing
Zijia Liu
Han, Peixuan
Yue Su
Haofei Yu
Jiaxuan You
Zijia Liu
Haofei Yu
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%