llm-sp

On a high level, the idea is similar to the . They train an LLM (called AdvPrompter) to automatically jailbreak a target LLM. AdvPrompter is trained on rewards (logprob of "Sure, here is...") of the target model. The result is good but maybe not as good as the at the time. However, there are a lot of interesting technical contributions

The authors also advocate for approximate memorization instead of verbatim, similar to

GitHub - iamgroot42/mimir: Python package for measuring memorization in LLMs.

Library of MIAs on LLMs, including Min-k%, zlib, reference-based attack (Ref), neighborhood

Temporal shift in member vs non-member test samples contributes to an overestimated MIA success rate. The authors measure this distribution shift with

Ask target LLM to select a verbatim text from a copyrighted book/ArXiv paper in a multiple-choice format (four choices). The other three options are close LLM-paraphrased texts. The core idea is similar to , but using MCQA instead of loss computation. The authors also debias/normalize for effects of the answer ordering, which LLMs are known to have trouble with

Propose SHIELD defense which works by (1) detecting copyrighted content in model’s output, (2) verifying it with internet search, and (3) few-shot prompting to let the model refuse or answer as appropriate (summary and QA are ok, but not verbatim). Defense seems very effective and is better than

The technique is based on

ChatGPT Plugin Exploit Explained: From Prompt Injection to Accessing Private Data [ ]

Advanced Data Exfiltration Techniques with ChatGPT [ ]

Hacking Google Bard - From Prompt Injection to Data Exfiltration [ ]

Securing LLM Systems Against Prompt Injection [ ]

Meme [ ]

https://github.com/corca-ai/awesome-llm-security

: A curation of awesome tools, documents and projects about LLM Security

: LLM security is the investigation of the failure modes of LLMs in use, the conditions that lead to them, and their mitigations

: CMSC818I: Advanced Topics in Computer Systems; Large Language Models, Security, and Privacy (UMD) by Prof. Yizheng Chen

: Crowd-sourced jailbreaks

https://github.com/ethz-spylab/rlhf_trojan_competition

: Competition track at SaTML 2024

https://github.com/Hannibal046/Awesome-LLM/

: Huge compilation of LLM papers and software

https://github.com/LostOxygen/llm-confidentiality

: Framework for evaluating LLM confidentiality

https://github.com/leondz/garak

: LLM vulnerability scanner

https://github.com/fiddler-labs/fiddler-auditor

: Fiddler Auditor is a tool to evaluate language models

https://github.com/NVIDIA/NeMo

: NeMo: a toolkit for conversational AI

is a method for bypassing safety filters, system instructions, or preferences. Sometimes asking the model directly (like prompt injection) does not work so more complex prompts (e.g., ) are used to trick the model