Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems
Надёжный агентный искусственный интеллект: режимы отказа, стратегии смягчения рисков и жизненный цикл автономных систем на основе больших языковых моделей
2026-09-19
SCID: 54.1/q9hdnufa
Discuss with AI
Trustworthy Agent Development Lifecycleagentic AIindirect prompt injectionlarge language modelsmemory contamination
Figures from the paper
Abstract (AI)
Agentic AI systems built on large language models can plan over multiple steps, use external tools, retain information in memory, and coordinate with other agents. These capabilities make them more useful than static language models, but they also introduce new security and operational risks. Untrusted content from websites, emails, documents, and databases can enter the same context as system instructions; persistent memory can carry compromised information across sessions; and access to external tools can turn an incorrect model response into a consequential real-world action. This article reviews the trustworthiness of agentic AI across five interconnected dimensions: safety and robustness, alignment and human oversight, transparency and auditability, privacy and data governance, and regulatory compliance. It organizes key failure modes, including indirect prompt injection, backdoor triggers, goal misgeneralization, memory contamination, and cross-session data leakage, into a unified taxonomy. It also examines major mitigation approaches, such as instruction hierarchies, context isolation, spotlighting, process-based supervision, constrained tool use, and privacy-preserving memory, while distinguishing techniques supported by empirical evidence from those that remain largely conceptual. Building on this analysis, we introduce the Trustworthy Agent Development Lifecycle (TADL), a six-phase framework covering specification, design, training, evaluation, deployment, and monitoring. For each phase, TADL identifies relevant trust activities, expected evidence, and risk-based decision gates. Although TADL has not yet been empirically validated, it provides a structured foundation for developing and evaluating more secure and accountable agentic systems. The article concludes by identifying gaps in current benchmarks and outlining priorities for future research.
Key Findings
1
Agentic LLM systems introduce distinct risks because untrusted content, persistent memory, and external tools can compromise instructions, propagate contamination, or trigger consequential actions.
2
The Trustworthy Agent Development Lifecycle (TADL) organizes specification, design, training, evaluation, deployment, and monitoring with phase-specific evidence requirements and risk-based decision gates; however, it has not yet been empirically validated.
3
The article presents a unified taxonomy covering failure modes such as indirect prompt injection, backdoor triggers, goal misgeneralization, memory contamination, and cross-session data leakage.
4
The review examines mitigation strategies including instruction hierarchies, context isolation, spotlighting, process-based supervision, constrained tool use, and privacy-preserving memory, distinguishing empirically supported methods from conceptual proposals.
5
Trustworthiness is evaluated across five dimensions: safety and robustness, alignment and human oversight, transparency and auditability, privacy and data governance, and regulatory compliance.
Research Object
agentic AI systems built on large language models (autonomous LLM systems)
Research Subject
trustworthiness, failure modes, mitigation strategies, and lifecycle governance across safety, alignment, transparency, privacy, and compliance
Publication Details
Publication Date
2026-09-19
Journal
Publisher
ISSN
Cited by
0
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai8
"Why Should I Trust You?": Explaining the Predictions of Any Classifier2016
Overcoming catastrophic forgetting in neural networks2017
Generative Agents: Interactive Simulacra of Human Behavior2023
Artificial Intelligence Risk Management Framework (AI RMF 1.0)2023
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection2023
Agentic AI: a comprehensive survey of architectures, applications, and future directions2025
A Survey on Trustworthy LLM Agents: Threats and Countermeasures2025
TRiSM for Agentic AI: A review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems2026