The numbers don't lie, but they do whisper. In the SADF study, fixing the model and varying the orchestration framework produced an attack success rate (ACR) spread from 11.9% to 31.1% — a 2.6× gap that most security assessments have been systematically ignoring. The assumption that model safety equals system safety is dead. The data is the witness.
Context: The Methodology Behind the Noise
The SADF team, led by Julie Brunias, presented at DEF CON 34 AI Village a forensic evaluation of four popular AI agent frameworks: CrewAI, LangChain, AutoGen, and SmolAgents, against a baseline of Direct API calls. They fixed the model to Claude Sonnet, controlled for prompt structure, and measured ACR over 5,119 evaluation rows with 32 adversarial payloads. The killer methodological move: they introduced a refusal-filtered scoring correction. Naive substring matching had been overestimating Claude’s security by 4–6×. After correction, Claude Sonnet’s real ACR dropped to 15.5%, and Claude Haiku to 22.3%. The study doesn’t just report numbers — it corrects the measurement error embedded in prior work.
Core: The On-Chain Evidence Chain
When I trace on-chain flows, I look for the moment the data deviates from the narrative. The SADF data does the same for agent security. The direct API baseline, representing a model without framework overhead, scored 15.5% ACR. CrewAI, with its discrete task isolation architecture, actually improved on that baseline at 11.9%. Then the curve steepens: LangChain at 18.1%, AutoGen at 20.0%, and SmolAgents at a staggering 31.1%.
The breakdown of failure modes tells the real story. The study catalogues eight categories: Tool Call Hijacking, Output Poisoning, Cross-Tool Injection, Memory Poisoning, RAG Poisoning, Delegated Authority Abuse, Multi-Agent Propagation, and Context Boundary Violation. SmolAgents alone showed a 64% Context Boundary Violation rate and a 20% RAG Poisoning rate — both unique to that framework. CrewAI’s low ACR came from its restricted tool scope, not from superior security design. The ledger of framework security is more nuanced than a single ranking.
Based on my experience auditing smart contract vulnerabilities during the 2022 collapse, I’ve learned that the most dangerous flaws hide in the composition layer, not the core logic. SADF’s work confirms this for AI agents. The framework is the attack surface. The model is just the fuel.
The study also provides a shared vocabulary for the industry. The eight failure modes are not just academic — they map to real CVEs. Azure SRE Agent (CVE-2026-62830) and Langflow (CVE-2026-9198) are concrete examples of framework-level vulnerabilities that have already been exploited in the wild. The data is not theoretical. The attack surface is real.
Contrarian: Correlation Is Not Causation
But here’s where the data detective’s instinct kicks in. The 32 payloads are a sample, not a census. They likely represent the researchers’ best guess at realistic attack vectors, but real-world adversarial payloads are infinitely more diverse. The study’s payloads may miss rare but high-impact paths. The simulation environment, while ethically necessary, strips away timing and permission complexity that real tool execution introduces. A framework that scores low in a sandbox might perform differently under real-world latency and permission chains.
Moreover, the study fixes the model to Claude Sonnet. Would the rankings hold with GPT-5.4, DeepSeek, or Llama? The model×framework interaction effect is unknown. The ACR of 31.1% for SmolAgents may be a property of the model’s refusal behavior as much as the framework’s architecture. The data is strong, but it is not final. The ledger of security is always being rewritten.
Another hidden variable: configuration. The study uses default settings for each framework. But in production, engineers tune system prompts, temperature, tool permissions, and memory limits. The sensitivity of ACR to these parameters is not decomposed. A framework with high default ACR might become secure under tight configuration, while a low-ACR framework might become vulnerable under lax configuration. The study provides a starting point, not a definitive purchasing guide.
Takeaway: The Next Signal
The next wave of security assessments will shift from model-level to system-level. The question is not whether your model is safe, but whether your framework-surrounding tool environment is safe. The SADF study is a POC, not a production toolkit, but it sets the direction. I expect to see security evaluation as a service (SEaaS) offerings emerge from firms like Palo Alto Unit 42 or CrowdStrike, embedding SADF-style metrics into CI/CD pipelines. The data is already there. The market will follow.
For now, the numbers are clear: frameworks are not neutral. CrewAI’s 11.9% ACR is not a seal of approval — it’s a baseline. SmolAgents’ 31.1% is not a death sentence — it’s a call for hardening. The ledger remembers everything. The architecture of your agent system is the attack surface. Act accordingly.
Following the attack surface, always.
On-chain evidence > Hype.
The ledger remembers everything.
Silence is suspicious.