SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
SafeScientist: к выявлению научных открытий с учетом рисков при использовании агентов LLM
2025-05-29
SCID: 54.1/vrv63j3b
Discuss with AI
AI scientist frameworkLLM agentsSafeScientistSciSafetyBenchadversarial attack robustnessagent-collaboration monitoringethical reviewerhigh-risk scientific tasksprompt monitoringsafety pipelinetool-related risk taskstool-use monitoring
Figures from the paper
Abstract (AI)
Recent advancements in large language model (LLM) agents have significantly accelerated scientific discovery automation, yet concurrently raised critical ethical and safety concerns. To systematically address these challenges, we introduce \textbf{SafeScientist}, an innovative AI scientist framework explicitly designed to enhance safety and ethical responsibility in AI-driven scientific exploration. SafeScientist proactively refuses ethically inappropriate or high-risk tasks and rigorously emphasizes safety throughout the research process. To achieve comprehensive safety oversight, we integrate multiple defensive mechanisms, including prompt monitoring, agent-collaboration monitoring, tool-use monitoring, and an ethical reviewer component. Complementing SafeScientist, we propose \textbf{SciSafetyBench}, a novel benchmark specifically designed to evaluate AI safety in scientific contexts, comprising 240 high-risk scientific tasks across 6 domains, alongside 30 specially designed scientific tools and 120 tool-related risk tasks. Extensive experiments demonstrate that SafeScientist significantly improves safety performance by 35\% compared to traditional AI scientist frameworks, without compromising scientific output quality. Additionally, we rigorously validate the robustness of our safety pipeline against diverse adversarial attack methods, further confirming the effectiveness of our integrated approach. The code and data will be available at https://github.com/ulab-uiuc/SafeScientist. \textcolor{red}{Warning: this paper contains example data that may be offensive or harmful.}
Key Findings
1
SafeScientist improves safety performance by 35% compared to traditional AI scientist frameworks while maintaining scientific output quality.
2
SafeScientist integrates multiple defensive mechanisms: prompt monitoring, agent-collaboration monitoring, tool-use monitoring, and an ethical reviewer component.
3
SafeScientist is an AI scientist framework that proactively refuses ethically inappropriate or high-risk scientific tasks.
4
SciSafetyBench is a new benchmark with 240 high-risk scientific tasks across 6 domains, 30 scientific tools, and 120 tool-related risk tasks for evaluating AI safety in scientific contexts.
5
The SafeScientist safety pipeline is robust against diverse adversarial attack methods, validating the effectiveness of the integrated approach.
Research Object
SafeScientist AI scientist framework for risk-aware scientific discovery (including its defensive components: prompt monitoring, agent-collaboration monitoring, tool-use monitoring, and ethical reviewer)
Research Subject
Improving safety and ethical responsibility in AI-driven scientific exploration by detecting/refusing high-risk or ethically inappropriate tasks and integrating multi-layer defensive mechanisms, evaluated via the SciSafetyBench benchmark and robustness tests against adversarial attacks
Publication Details
Publication Date
2025-05-29
Journal
Publisher
ISSN
Cited by
2
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest