LLM Jailbreak Defenses: Why Static Filters Fail

Almost every LLM jailbreak defense on the market reports a strong number. Detection rates in the high nineties, single-digit attack success rates, benchmark charts with a satisfying gap between the guarded and unguarded bars. Then the same defense gets deployed and something that took a determined person an afternoon walks straight through it.

The gap is not vendor dishonesty in most cases. It is a methodological problem with how jailbreak defenses are evaluated, and it has now been measured directly. This is an analysis of what the published evaluations show, why the standard methodology overstates robustness, and what a defense evaluation has to do differently to produce a number worth trusting.

The static evaluation trap

The default way to evaluate a jailbreak defense is to assemble a set of known attack prompts, run them through the guarded system, and report how many were blocked. It is easy to run, easy to automate, and it measures almost the wrong thing.

A fixed prompt set measures robustness against attacks that already existed when the defense was built. Every defense is, implicitly or explicitly, trained or tuned on the same public corpora that supply the evaluation prompts — AdvBench, HarmBench, JailbreakBench and their derivatives. Scoring well on that distribution is the design target, not an independent test of it.

Real attackers do not sample from a fixed set. They look at the defense, work out what it keys on, and construct something it does not key on. The technical term is an adaptive attack, and it is the only threat model that matches how the attack actually happens. NIST AI 100-2 E2025 makes the same point structurally when it defines attacker capability and knowledge as a required dimension of any adversarial ML threat model: a mitigation claim without a stated attacker capability is not a claim about security.

What happens when the attacker is allowed to adapt

The 2025 paper The Attacker Moves Second, from a team spanning OpenAI, Anthropic, Google DeepMind and ETH Zurich, is the most direct measurement of this gap currently available. The authors took twelve recently published defenses against jailbreaks and prompt injection, then attacked each one adaptively rather than with a fixed prompt set, using gradient descent, reinforcement learning, random search and human-guided exploration, and spending real compute per target.

The reported result: most of the twelve fell to attack success rates above 90%. Search-based adaptive attacks exceeded 90% ASR against Protect AI Detector, PromptGuard and Model Armor, with PIGuard at 71%. These are not obscure research prototypes; several are shipped products.

The important detail is that these defenses were not badly built. They were evaluated against static attack sets or against computationally weak optimization, published a strong number in good faith, and that number did not survive an attacker who was permitted to look at the defense first.

Defense families and how each one is bypassed

The bypasses are not mysterious once the keying signal of each family is named.

Defense family	What it keys on	The adaptive counter
Input classifier / prompt filter	Learned features of known attack text	Optimize a prompt that keeps semantics and moves off the classifier’s features
Perplexity filter	Statistically improbable token runs	Switch to fluent optimized prompts (AutoDAN-class)
Paraphrase / smoothing defense	Fragility of a specific adversarial string	Attacks robust to perturbation, or semantic rather than token-level attacks
System-prompt hardening	Explicit instruction to refuse	Multi-turn escalation and payload splitting, which never state the request
Output classifier	Harmful content in the response	Fragmented or encoded output, or requests for structure rather than content
Secondary-model judge	The judge’s own alignment	The judge is an LLM and is itself jailbreakable
Per-message policy check	One message at a time	Conversation-level attacks, where every single message is benign

Two rows deserve emphasis, because both failure modes are architectural rather than tuning problems.

Perplexity filtering is the clearest case of a defense that worked exactly until someone wrote the counter. It reliably catches gradient-optimized adversarial suffixes, whose token runs are statistically bizarre, and it does nothing at all against genetic-search methods that optimize for fluent text. One defense, one paper, obsolete. The mechanics of both attack styles are covered in the GCG practitioner guide.

Per-message policy checks fail by construction against multi-turn attacks. If a defense scores each user message independently, a Crescendo-style escalation is invisible to it, because no individual message in the sequence is objectionable. No threshold setting fixes that; the defense is scoring the wrong unit. Crescendo’s authors report high attack success rates across every model and task they evaluated, and human-guided exploration is one of the four attack strategies The Attacker Moves Second used to break its twelve targets. Several of the attack patterns a per-message check misses are catalogued in the documented jailbreak examples.

The metric is also part of the problem

Even before adaptivity, reported robustness numbers are less stable than they look. The Bag of Tricks benchmark systematically varied eight key factors in jailbreak evaluation — target model, system prompt presence, judge model, evaluation dataset, attack budget and others — across roughly 354 experiments and about 55,000 A800-80G GPU hours, and found that measured effectiveness moves substantially with choices that papers frequently report in a footnote or not at all.

For a defender reading vendor material, three questions separate a meaningful number from a decorative one:

• What judged success? A keyword matcher and a strict model grader produce very different ASR values on identical outputs.
• Was there a system prompt? Evaluating a bare model overstates the exposure of a hardened deployment; evaluating a hardened one understates the risk of an exposed endpoint.
• What was the attacker’s budget? Best-of-N style attacks show success scaling predictably with sample count, so any ASR quoted without a query budget is incomplete by construction.

A defense that cannot answer those three has not been evaluated, it has been demonstrated.

What actually holds up

The picture is not uniformly bleak, and the contrast is instructive.

Anthropic’s Constitutional Classifiers work is the counterexample worth studying, mostly because of its methodology. Rather than reporting performance against a fixed prompt set, the evaluation ran an extended human red-teaming programme: over 3,000 estimated hours, during which no red teamer found a universal jailbreak that extracted information from the classifier-guarded model at a level of detail comparable to an unguarded model across most target queries. The published deployment cost is also stated rather than elided: an absolute 0.38% increase in production-traffic refusals and 23.7% inference overhead.

Three properties distinguish that result from the twelve defenses that fell:

1. The attacker was adaptive and well resourced. Thousands of hours of human red teaming with knowledge of the defense, not a static corpus replay.
2. The claim is bounded. “No universal jailbreak was found across most target queries under this budget” is a falsifiable statement with a threat model attached. “Blocks 99% of attacks” is not.
3. The cost is disclosed. Refusal rate and latency overhead are the numbers that decide whether a defense survives contact with a product team, and defenses that hide them tend to get switched off in production.

The general lesson is that jailbreak defense is an economic control, not a boolean one. None of the published work supports the claim that any current defense makes jailbreaks impossible. What good defenses do is raise the cost per successful attack — more queries, more compute, more human effort — which is only valuable if the surrounding system actually charges that cost. Rate limiting, per-identity anomaly detection and query-budget caps are therefore part of the jailbreak defense, not adjacent hygiene. A defense that multiplies the required query count by a hundred is worthless in front of an unmetered endpoint.

Evaluating your own stack honestly

A defensible internal evaluation looks different from a vendor benchmark:

• Assume the attacker knows the defense. Evaluate with the guardrail configuration, system prompt and filter thresholds visible to the tester. Security through undisclosed prompt text does not survive one leaked system prompt.
• Give the attack a budget and state it. Report ASR at a fixed number of queries per behaviour. Curves beat point estimates, because a defense that flattens at high budget is genuinely different from one that merely starts lower.
• Score conversations, not messages. Any evaluation that submits single-turn prompts will miss the entire multi-turn family, which is where most current production failures live.
• Hold out a private set. Public benchmark prompts leak into training data and into filter tuning. Keeping a private behaviour set is the only way to detect that a score improved because the filter memorised the test.
• Re-run it on every model or prompt change. Robustness is not a property of the guardrail alone; it is a property of the guardrail plus the model plus the system prompt. Changing any of the three invalidates the previous number, which is the argument for wiring this into CI rather than running it at release. The prompt injection CI gate walkthrough covers the mechanics of running that as a regression suite.

The detection side deserves the same scrutiny. The four detection approaches ranked sets out what each signal class can see and where each one is blind, which is the input to deciding how many independent layers a deployment actually needs.

The practical position

Treat every published jailbreak defense number as an upper bound measured under favourable conditions, and assume the operational figure is materially worse. Build for defense in depth rather than for a single filter that is claimed to work: input screening, conversation-level scoring, output screening, strict privilege separation on tools, and metering that makes high-volume search attacks expensive.

Then scope the testing against the architecture rather than against a generic checklist. The Red Team Plan Builder returns an ordered engagement checklist for the components and trust boundaries a specific system actually has, which is the shortest route from this analysis to a test plan. For the attack side of the same ground, the jailbreak techniques overview covers the technique families and many-shot jailbreaking covers the long-context case that most per-message filters miss entirely. Defensive tooling and guardrail architecture are covered in more depth at aidefense.dev.