Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Не то, на что вы подписывались: компрометация реальных приложений, интегрированных с большими языковыми моделями, с помощью косвенного внедрения промптов
2023-11-21
SCID: 54.1/fexpaqar
Discuss with AI
LLM-integrated applicationsarbitrary code executiondata theftindirect prompt injectioninformation ecosystem contamination
Figures from the paper
Abstract (AI)
Large Language Models (LLMs) are increasingly being integrated into applications, with versatile functionalities that can be easily modulated via natural language prompts. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We show that LLM-Integrated Applications blur the line between data and instructions and reveal several new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (i.e., without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved at inference time. We derive a comprehensive taxonomy from a computer security perspective to broadly investigate impacts and vulnerabilities, including data theft, worming, information ecosystem contamination, and other novel security risks. We then demonstrate the practical viability of our attacks against both real-world systems, such as Bing Chat and code-completion engines, and GPT-4 synthetic applications. We show how processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and if other APIs are called. Despite the increasing reliance on LLMs, effective mitigations of these emerging threats are lacking. By raising awareness of these vulnerabilities, we aim to promote the safe and responsible deployment of these powerful models and the development of robust defenses that protect users from potential attacks.
Key Findings
1
Existing mitigations are described as ineffective or insufficient despite growing reliance on LLM-integrated applications.
2
Experiments demonstrate practical attacks against Bing Chat, code-completion engines, and GPT-4-based synthetic applications.
3
Indirect prompt injection enables remote attacks on LLM-integrated applications by embedding malicious instructions in data retrieved during inference.
4
Retrieved prompts can function as arbitrary code execution, alter application behavior, and control whether and how external APIs are invoked.
5
The study develops a security-oriented taxonomy covering impacts including data theft, self-propagating attacks, information ecosystem contamination, and other risks.
Research Object
real-world LLM-integrated applications
Research Subject
vulnerabilities and security impacts caused by indirect prompt injection, including remote exploitation, data theft, functionality manipulation, arbitrary code execution, and API-call control
Publication Details
Publication Date
2023-11-21
Journal
Publisher
ISSN
Cited by
652
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
Cited by19
Unleashing the potential of prompt engineering for large language models2025
Large Language Models for Cyber Security: A Systematic Literature Review2025
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges2025
Combating misinformation in the age of LLMs: Opportunities and challenges2024
Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects2023
SoK: Agentic Skills -- Beyond Tool Use in LLM Agents2026
Optimization-based Prompt Injection Attack to LLM-as-a-Judge2024
Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges2026
A Survey on Trustworthy LLM Agents: Threats and Countermeasures2025
AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems2026
Risks of AI scientists: prioritizing safeguarding over autonomy2025
TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems2026
LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks2026
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security2026
A Comparative Survey of Security Risks in AI Systems: From LLMs to AI Agents and Embodied Agents2026
Agentic AI and Autonomous Systems2026
CapChain: A Capability-Token Access Control Architecture with Verifiable Provenance for Multi-Agent LLM Systems2026
Trustworthy Agentic AI: Failure Modes, Mitigation Strategies, and a Lifecycle Framework for Autonomous LLM Systems2026
From AI-generated content to agentic action: Security and safety threats in generative AI2026