Risks of AI scientists: prioritizing safeguarding over autonomy
Риски «AI-учёных»: приоритет защиты над автономией
2025-09-18
SCID: 54.1/7uadmjxc
Discuss with AI
AI scientistsagent alignmenthuman regulationlarge language modelssafeguarding vs autonomy
Figures from the paper
Abstract (AI)
AI scientists powered by large language models have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines. While their capabilities are promising, these agents also introduce novel vulnerabilities that require careful consideration for safety. However, there has been limited comprehensive exploration of these vulnerabilities. This perspective examines vulnerabilities in AI scientists, shedding light on potential risks associated with their misuse, and emphasizing the need for safety measures. We begin by providing an overview of the potential risks inherent to AI scientists, taking into account user intent, the specific scientific domain, and their potential impact on the external environment. Then, we explore the underlying causes of these vulnerabilities and provide a scoping review of the limited existing works. Based on our analysis, we propose a triadic framework involving human regulation, agent alignment, and an understanding of environmental feedback (agent regulation) to mitigate these identified risks. Furthermore, we highlight the limitations and challenges associated with safeguarding AI scientists and advocate for the development of improved models, robust benchmarks, and comprehensive regulations.
Key Findings
1
A triadic mitigation framework is proposed: human regulation, agent alignment, and understanding environmental feedback (agent regulation).
2
AI scientists powered by large language models can autonomously conduct experiments and facilitate scientific discoveries across disciplines.
3
Risks depend on user intent, scientific domain, and potential impact on the external environment.
4
There are significant limitations and challenges to safeguarding AI scientists, necessitating better models, robust benchmarks, and comprehensive regulations.
5
These AI scientists introduce novel vulnerabilities and potential for misuse that require careful safety consideration.
6
Underlying causes of vulnerabilities are analyzed and existing literature on these vulnerabilities is limited.
Research Object
AI scientists powered by large language models
Research Subject
Vulnerabilities and risks associated with autonomous AI scientists, including misuse, underlying causes, and mitigation via human regulation, agent alignment, and environmental feedback (safeguarding over autonomy)
Publication Details
Publication Date
2025-09-18
Journal
Publisher
ISSN
Cited by
51
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai10
A Survey on Evaluation of Large Language Models2024
Large language models encode clinical knowledge2023
A survey on large language model based autonomous agents2024
Generative Agents: Interactive Simulacra of Human Behavior2023
Autonomous chemical research with large language models2023
Augmenting large language models with chemistry tools2024
Self-Driving Laboratories for Chemistry and Materials Science2024
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection2023
Empowering biomedical discovery with AI agents2024
A review of large language models and autonomous agents in chemistry2024
Cited by7
Empowering biomedical discovery with AI agents2024
A review of large language models and autonomous agents in chemistry2024
A Survey on Trustworthy LLM Agents: Threats and Countermeasures2025
Autonomous in-silico inorganic materials discovery via multi-agent physics-aware scientific reasoning2026
Agentic AI in Healthcare: Opportunities, Challenges, and Future Directions2026
A Comparative Survey of Security Risks in AI Systems: From LLMs to AI Agents and Embodied Agents2026
AI-Generated Hypotheses and the Emergence of Autonomous Scientific Discovery2026