Prompt Injection attack against LLM-integrated Applications
Атаки с внедрением промптов на приложения, интегрированные с большими языковыми моделями
2023-06-08
SCID: 54.1/fbujx5yy
Discuss with AI
HouYiLLM-integrated applicationsapplication prompt theftblack-box attackprompt injection attacks
Figures from the paper
Abstract (AI)
Large Language Models (LLMs), renowned for their superior proficiency in language comprehension and generation, stimulate a vibrant ecosystem of applications around them. However, their extensive assimilation into various services introduces significant security risks. This study deconstructs the complexities and implications of prompt injection attacks on actual LLM-integrated applications. Initially, we conduct an exploratory analysis on ten commercial applications, highlighting the constraints of current attack strategies in practice. Prompted by these limitations, we subsequently formulate HouYi, a novel black-box prompt injection attack technique, which draws inspiration from traditional web injection attacks. HouYi is compartmentalized into three crucial elements: a seamlessly-incorporated pre-constructed prompt, an injection prompt inducing context partition, and a malicious payload designed to fulfill the attack objectives. Leveraging HouYi, we unveil previously unknown and severe attack outcomes, such as unrestricted arbitrary LLM usage and uncomplicated application prompt theft. We deploy HouYi on 36 actual LLM-integrated applications and discern 31 applications susceptible to prompt injection. 10 vendors have validated our discoveries, including Notion, which has the potential to impact millions of users. Our investigation illuminates both the possible risks of prompt injection attacks and the possible tactics for mitigation.
Key Findings
1
An exploratory analysis of ten commercial LLM-integrated applications exposed practical limitations in existing prompt injection attack strategies.
2
HouYi enables severe attack outcomes, including unrestricted arbitrary LLM usage and straightforward theft of application prompts.
3
HouYi is introduced as a black-box prompt injection technique combining a pre-constructed prompt, context-partitioning injection prompt, and malicious payload.
4
Ten vendors validated the reported vulnerabilities, including Notion, potentially affecting millions of users; the study also discusses mitigation tactics.
5
Testing 36 real-world LLM-integrated applications found 31 susceptible to prompt injection attacks.
Research Object
actual LLM-integrated applications
Research Subject
vulnerability of these applications to prompt injection attacks and the resulting security impacts
Publication Details
Publication Date
2023-06-08
Journal
Publisher
ISSN
Cited by
988
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
Cited by15
Unleashing the potential of prompt engineering for large language models2025
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly2024
Large Language Models for Software Engineering: A Systematic Literature Review2024
Large Language Models for Cyber Security: A Systematic Literature Review2025
Combating misinformation in the age of LLMs: Opportunities and challenges2024
AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways2025
Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects2023
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents2024
A Survey on Trustworthy LLM Agents: Threats and Countermeasures2025
AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems2026
TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems2026
LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks2026
AI agents in drug discovery: applications and case studies2026
Agentic AI in Healthcare: Opportunities, Challenges, and Future Directions2026
A Comparative Survey of Security Risks in AI Systems: From LLMs to AI Agents and Embodied Agents2026