InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
InjecAgent: бенчмаркинг косвенных внедрений промптов в агентных системах на основе больших языковых моделей с интеграцией инструментов
2024-01-01
SCID: 54.1/z8fpatwt
Discuss with AI
INJECAGENT benchmarkattack success rateindirect prompt injectionprivate data exfiltrationtool-integrated LLM agents
Figures from the paper
Abstract (AI)
Recent work has embodied LLMs as agents, allowing them to access tools, perform actions, and interact with external content (e.g., emails or websites).However, external content introduces the risk of indirect prompt injection (IPI) attacks, where malicious instructions are embedded within the content processed by LLMs, aiming to manipulate these agents into executing detrimental actions against users.Given the potentially severe consequences of such attacks, establishing benchmarks to assess and mitigate these risks is imperative.In this work, we introduce INJECAGENT, a benchmark designed to assess the vulnerability of tool-integrated LLM agents to IPI attacks.INJECAGENT comprises 1,054 test cases covering 17 different user tools and 62 attacker tools.We categorize attack intentions into two primary types: direct harm to users and exfiltration of private data.We evaluate 30 different LLM agents and show that agents are vulnerable to IPI attacks, with ReAct-prompted GPT-4 vulnerable to attacks 24% of the time.Further investigation into an enhanced setting, where the attacker instructions are reinforced with a hacking prompt, shows additional increases in success rates, nearly doubling the attack success rate on the ReAct-prompted GPT-4.Our findings raise questions about the widespread deployment of LLM Agents.
Key Findings
1
Attack intentions are categorized into direct harm to users and exfiltration of private data.
2
Evaluation of 30 LLM agents demonstrates broad vulnerability to indirect prompt injection attacks.
3
INJECAGENT introduces a benchmark for evaluating indirect prompt injection vulnerabilities in tool-integrated LLM agents.
4
ReAct-prompted GPT-4 was successfully attacked in 24% of cases, with success nearly doubling when attacker instructions included a reinforcing hacking prompt.
5
The benchmark contains 1,054 test cases spanning 17 user tools and 62 attacker tools.
Research Object
tool-integrated large language model agents exposed to indirect prompt injection attacks through external content
Research Subject
vulnerability to indirect prompt injection attacks, including harmful action execution and private-data exfiltration
Publication Details
Publication Date
2024-01-01
Journal
Publisher
ISSN
Cited by
143
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai1
Cited by10
Unleashing the potential of prompt engineering for large language models2025
When LLMs meet cybersecurity: a systematic literature review2025
AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways2025
LLM-Based Agents for Tool Learning: A Survey2025
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges2026
A Survey on Trustworthy LLM Agents: Threats and Countermeasures2025
TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems2026
A Comparative Survey of Security Risks in AI Systems: From LLMs to AI Agents and Embodied Agents2026
SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration2026
From AI-generated content to agentic action: Security and safety threats in generative AI2026