AI Red Teaming Grew Up. Its Measurements Didn't.

The field finally agrees the agent is the attack surface. But we're measuring it with the same broken yardstick we just spent three years learning to distrust in the chat window.

There is a rare moment of consensus in AI security right now, and it is worth pausing on. Ask any serious red teamer where the risk lives in 2026 and you will hear the same answer: the agent, not the chatbot. Indirect prompt injection — a payload hidden in a retrieved document, a tool's output, an MCP server's response, another agent's message — is "the new SQL injection," a layer-violation where data quietly becomes instructions. OWASP has ranked prompt injection first on its LLM Top 10 two years running and published a dedicated Agentic Top 10, led by goal hijacking and tool misuse. The surface is settled. The vocabulary is settled.

What is not settled — and what almost no vendor deck will tell you — is that most of the numbers being produced against that surface are close to meaningless. Not because the tools are bad, but because of how they are run. We are repeating, in the agentic layer, the single most expensive measurement mistake of the chat-jailbreak era. And we are doing it while congratulating ourselves for having moved on from it.

The lesson the chat era paid for in full

Rewind to the jailbreak wars. For a stretch, the standard way to claim a model or a defense was "robust" was to fire a fixed battery of automated attacks at it once and report the attack success rate. The numbers were reassuring. They were also a lie of omission — and the field discovered this the hard way. As one 2026 survey of frontier-lab practice puts it bluntly: "single-turn automated attacks like AutoDAN, GCG, and PAIR yield reassuringly low attack success rates while multi-turn human red teaming exposes failures up to 75% ASR."

The adaptive methods that followed made the point unarguable. Microsoft's Crescendo escalates a conversation across turns and beats single-shot methods by 49–71% on Gemini-Pro. Tree-of-Attacks-with-Pruning jailbreaks GPT-4-class models on more than 80% of prompts with black-box access alone, using fewer queries than what came before. Many-shot jailbreaking succeeds at 256 examples where five did nothing. The through-line is not "attacks got stronger." It is that a static score measures the target against the one payload you happened to try, not against an adversary who adapts. Microsoft's own summary of red-teaming a hundred products lands on the same discipline: AI red teaming is not safety benchmarking, and any evaluation is a lower bound, never a certificate.

A static attack success rate is not a measurement of safety. It is a measurement of your imagination.

Now watch us do it again, with agents

Here is the part that should worry practitioners. The agentic benchmarks are excellent — AgentDojo, AgentHarm, SHADE-Arena give us realistic tool suites and ground-truth success oracles. But watch how they are typically used in a security review: pick a suite, run one crafted injection per task, once, read the aggregate, write "robust." Against a strong 2026 model, that static injection often scores at or near zero. The report ships. Everyone exhales.

In our own controlled reproduction, we pointed a standard agentic banking suite at three frontier models with a strong fixed injection. Every one scored 0%. Had we stopped there — as most reviews do — the finding would have read "robust." So we didn't stop. We replaced the fixed payload with a lightweight adaptive loop: run the agent, read its trace, and if the injection missed, let an attacker model rewrite it and try again within a handful of queries. On the identical suite, the saturated zero turned into real, ground-truth-confirmed breaches on the most susceptible model.

Same suite, same tasks, same model an attacker allowed to read the trace and iterate. The number that reflects a real adversary. None of this is a novel discovery on my part two 2026 papers (AutoDojo and IterInject) made the same point earlier and with more rigor, and the field has already observed that "leading LLMs are surprisingly compliant with malicious instructions in agent settings even without jailbreaking." Treat my run as a confirmation, not a claim. But that is precisely why it matters: if a casual reproduction flips 0% to a third of cells, the static agent-injection score deserves exactly as much trust as the static chat-jailbreak score we already learned to throw away. Which is to say: almost none.

The three questions that turn a score into evidence

So how do you read a red-team number — your own team's, or a vendor's — and know whether it means anything? Three questions. If a report cannot answer all three, its headline figure is decoration.

Q1 — ADAPTIVITY & BUDGET

Was the attacker allowed to adapt, and how many queries did it get?

"We block 99%" against a fixed payload is a statement about that payload, not about your system. Demand the attacker's feedback loop and its query budget. A number with no stated budget is a number with no adversary behind it.

Q2 — THE GRADER

Who judged success, and what is the judge's measured accuracy?

Automated red teaming leans on an LLM judge, and grading harm is "an open methodological problem" — inter-annotator agreement is genuinely hard. Worse, teams routinely let a model grade itself: in one of our runs a model scored 100% on an attack class while acting as its own judge. Ask for the judge's gold-set accuracy, its agreement (κ) with an independent grader, and confirm the judge is not the target.

Q3 — THE TARGET LAYER

Did you test the model, or the agent?

A model that refuses harm in a chat box will still execute it as an agent when the instruction arrives on a channel it trusts — a tool result, a retrieved doc, its own memory. Chat-layer safety and agent-layer safety are different measurements. If the red team only exercised the prompt, it never touched the surface where the impact lives.

There is a fourth trap hiding underneath all three: contamination. Meta's Muse Spark report found 19.8% evaluation-awareness on public benchmarks versus 2.0% on internal ones — a tenfold signal leak that, in their words, "breaks the assumption underneath every public adversarial leaderboard." Models increasingly know when they are being tested. A public-benchmark score is not just a lower bound; it may be a performed one.

What a rigorous program does instead

The frontier labs have already converged on the answer, and it is not "buy more scanners." Rigor looks like this:

• Private, adaptive suites over public scores. Build a domain-expert adversarial set for your actual deployment, refresh it at a cadence matched to risk, and let the attacker iterate. Public leaderboards are for orientation, not assurance.

• Grade the grader. Calibrate your judge against a human-labeled gold set, report its accuracy and cross-judge agreement, and never let the target score itself.

• Aim at the data path. Grade agentic trajectories — tool selection, recovery, leakage — at each step, and inject on the channels an agent trusts, not just the user prompt.

• Buy provenance, not more filters. The controls that actually hold are structural: control/data-flow separation and capability-based least privilege (the CaMeL line of work), signed provenance on memory and retrieval. A hijacked step that holds no dangerous capability has nothing to exploit.

• Run it continuously. Wire the adaptive suite into CI so every fine-tune and data refresh re-earns the number. Map coverage to the OWASP Agentic Top 10 and document it for EU AI Act Article 55 and NIST AI 600-1 — but treat the framework as a checklist for breadth, never as evidence of depth.

  
  

The takeaway

• The surface moved to the agent; the yardstick didn't. Static agent-injection scores under-report exactly the way static chat-jailbreak scores did.

• Adaptivity, the grader, and the target layer are the three questions that decide whether a red-team number is evidence or decoration.

• Public benchmark scores are contaminated by evaluation-awareness — a tenfold signal leak. Use private, adaptive, continuously-run suites.

• A red-team result is a lower bound you keep pushing down, not a certificate you frame on the wall.

  
  

The maturity of a red-team program is not measured by how low its attack success rate is. A low number is the easiest thing in the world to manufacture — run a weak attacker, once, and let a friendly judge grade it. Maturity is measured by how hard the program worked to make that number go up, and how honestly it reports what it could not break. In the agentic era, with real capabilities wired to real tools and real money, that honesty is not an academic nicety. It is the difference between a number that protects you and a number that only comforts you.