From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy
От ChatGPT к ThreatGPT: влияние генеративного искусственного интеллекта на кибербезопасность и конфиденциальность
2023-01-01
SCID: 54.1/6fcfb9vy
Discuss with AI
Generative AIcybersecurity and privacymalware detectionprompt injection attackssocial engineering attacks
Figures from the paper
Abstract (AI)
Undoubtedly, the evolution of Generative AI (GenAI) models has been the highlight of digital transformation in the year 2022. As the different GenAI models like ChatGPT and Google Bard continue to foster their complexity and capability, it’s critical to understand its consequences from a cybersecurity perspective. Several instances recently have demonstrated the use of GenAI tools in both the defensive and offensive side of cybersecurity, and focusing on the social, ethical and privacy implications this technology possesses. This research paper highlights the limitations, challenges, potential risks, and opportunities of GenAI in the domain of cybersecurity and privacy. The work presents the vulnerabilities of ChatGPT, which can be exploited by malicious users to exfiltrate malicious information bypassing the ethical constraints on the model. This paper demonstrates successful example attacks like Jailbreaks, reverse psychology, and prompt injection attacks on the ChatGPT. The paper also investigates how cyber offenders can use the GenAI tools in developing cyber attacks, and explore the scenarios where ChatGPT can be used by adversaries to create social engineering attacks, phishing attacks, automated hacking, attack payload generation, malware creation, and polymorphic malware. This paper then examines defense techniques and uses GenAI tools to improve security measures, including cyber defense automation, reporting, threat intelligence, secure code generation and detection, attack identification, developing ethical guidelines, incidence response plans, and malware detection. We will also discuss the social, legal, and ethical implications of ChatGPT. In conclusion, the paper highlights open challenges and future directions to make this GenAI secure, safe, trustworthy, and ethical as the community understands its cybersecurity impacts.
Key Findings
1
ChatGPT vulnerabilities can be exploited to bypass ethical safeguards and exfiltrate malicious information through jailbreaks, reverse psychology, and prompt injection attacks.
2
Cyber offenders may use generative AI to support social engineering, phishing, automated hacking, attack-payload generation, malware creation, and polymorphic malware development.
3
Generative AI can strengthen cyber defenses through automation, threat intelligence, secure code generation and detection, attack identification, incident response, and malware detection.
4
Generative AI creates both defensive cybersecurity opportunities and offensive risks, with significant social, ethical, and privacy implications.
5
The paper identifies unresolved challenges requiring ethical guidelines and further research to make generative AI secure, safe, trustworthy, and ethical.
Research Object
Generative AI models (exemplified by ChatGPT and Google Bard) as used in cybersecurity and privacy contexts
Research Subject
The cybersecurity, privacy, social, legal, and ethical impacts, vulnerabilities, attack uses, and defensive applications of Generative AI
Publication Details
Publication Date
2023-01-01
Journal
Publisher
ISSN
Cited by
1004
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai3
Cited by12
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly2024
When LLMs meet cybersecurity: a systematic literature review2025
Prompt Engineering in Medical Education2023
AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways2025
Conversational and generative artificial intelligence and human–chatbot interaction in education and research2024
Application of Large Language Models in Cybersecurity: A Systematic Literature Review2024
Artificial intelligence (AI) cybersecurity dimensions: a comprehensive framework for understanding adversarial and offensive AI2024
Generative AI and LLMs for Critical Infrastructure Protection: Evaluation Benchmarks, Agentic AI, Challenges, and Opportunities2025
LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks2026
A Survey on Human-AI Collaboration with Large Foundation Models2026
Integrating artificial intelligence capabilities in supply chain cyber risk management2026
Morality Meets Machine: How Evaluator and Agent Gender Shape Human-AI Moral Judgment2026