awesome-llm-security

"Visual Adversarial Examples Jailbreak Large Language Models", 2023-06, AAAI(Oral) 24, ,

"Are aligned neural networks adversarially aligned?", 2023-06, NeurIPS(Poster) 23, ,

"(Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs", 2023-07,

"Universal and Transferable Adversarial Attacks on Aligned Language Models", 2023-07, ,

"Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models", 2023-07, ,

"Image Hijacking: Adversarial Images can Control Generative Models at Runtime", 2023-09, ,

"Weak-to-Strong Jailbreaking on Large Language Models", 2024-04, ,

"Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection", 2023-02, AISec@CCS 23

"Jailbroken: How Does LLM Safety Training Fail?", 2023-07, NeurIPS(Oral) 23,

"Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models", 2023-07,

"Effective Prompt Extraction from Language Models", 2023-07, ,

"Multi-step Jailbreaking Privacy Attacks on ChatGPT", 2023-04, EMNLP 23, ,

"LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?", 2023-07,

"Jailbreaking chatgpt via prompt engineering: An empirical study", 2023-05,

"Prompt Injection attack against LLM-integrated Applications", 2023-06,

"MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots", 2023-07, ,

"GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher", 2023-08, ICLR 24, ,

"Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities", 2023-08,

"Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs", 2023-08,

"Detecting Language Model Attacks with Perplexity", 2023-08,

"Open Sesame! Universal Black Box Jailbreaking of Large Language Models", 2023-09, ,

"Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!", 2023-10, ICLR(oral) 24,

"AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models", 2023-10, ICLR(poster) 24, , ,

"Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations", 2023-10, CoRR 23, ,

"Multilingual Jailbreak Challenges in Large Language Models", 2023-10, ICLR(poster) 24,

"Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation", 2023-11, SoLaR(poster) 24,

"DeepInception: Hypnotize Large Language Model to Be Jailbreaker", 2023-11,

"A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily", 2023-11, NAACL 24,

"AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models", 2023-10,

"Language Model Inversion", 2023-11, ICLR(poster) 24,

"An LLM can Fool Itself: A Prompt-Based Adversarial Attack", 2023-10, ICLR(poster) 24,

"GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts", 2023-09,

"Many-shot Jailbreaking", 2024-04,

"Rethinking How to Evaluate Language Model Jailbreak", 2024-04,

"BITE: Textual Backdoor Attacks with Iterative Trigger Injection", 2022-05, ACL 23,

"Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models", 2023-05, EMNLP 23,

"Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection", 2023-07, NAACL 24,

"Instructional Fingerprinting of Large Language Models", 2024-01, NAACL 24

"TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification", 2024-02, ACL 24 (findings)

"LLMmap: Fingerprinting For Large Language Models", 2024-07,

"Baseline Defenses for Adversarial Attacks Against Aligned Language Models", 2023-09,

"LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked", 2023-08, ICLR 24 Tiny Paper, ,

"Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM", 2023-09, ,

"Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models", 2023-12,

"AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks", 2024-03,

"Protecting Your LLMs with Information Bottleneck", 2024-04,

"PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetition", 2024-05, ICML 24,

“Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs”, 2024-06,

"Improving Alignment and Robustness with Circuit Breakers", 2024-06, NeurIPS 24, ,

"LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI’s ChatGPT Plugins", 2023-09,

"Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks", 2023-10, ACL 24,

"Security and Privacy Challenges of Large Language Models: A Survey", 2024-02,

"Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models", 2024-03,

"Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)", 2024-07,

"JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models", 2024-03,

"AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents", 2024-06, NeurIPS 24,

"AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents", 2024-10,

Plexiglass

: a security toolbox for testing and safeguarding LLMs

PurpleLlama

: set of tools to assess and improve LLM security

Rebuff

: a self-hardening prompt injection detector

Garak

: a LLM vulnerability scanner

LLMFuzzer

: a fuzzing framework for LLMs

LLM Guard

: a security toolkit for LLM Interactions

Vigil

: a LLM prompt injection detection toolkit

jailbreak-evaluation

: an easy-to-use Python package for language model jailbreak evaluation

Prompt Fuzzer

: the open-source tool to help you harden your GenAI applications

WhistleBlower

: open-source tool designed to infer the system prompt of an AI agent based on its generated text outputs

(0din GenAI Bug Bounty from Mozilla)( ): The 0Day Investigative Network is a bug bounty program focusing on flaws within GenAI models. Vulnerability classes include Prompt Injection, Training Data Poisoning, DoS, and more

: a prompt injection wargame

: a prompt injection aware chatbot designed to mitigate adversarial efforts

LLM Security Problems at DEFCON31 Quals

: the world's top security competition

Twitter:

Blog: authored by

Blog:

Blog:

Newsletter:

Newsletter & Blog: