Why Eddie Oz's 'LLMs Under Siege' Is the Defensive Wake-Up Call AI Security Needed

A response from the author of the redteam-ai-benchmark framework on what 30 tested models reveal about the state of AI security in 2026.

Introduction

In June 2026, Edilson Osorio Jr. (Eddie Oz) published "LLMs Under Siege: The Red Team Reality Check of 2026" — a comprehensive analysis that subjected 30 distinct AI models to real-world offensive security scenarios using the redteam-ai-benchmark framework.

As the author of that benchmark, I want to highlight why Eddie's work stands out as exactly the kind of defensive research the AI security community needs right now. This is not about celebrating model capabilities — it is about measuring exposure so defenders can act before attackers do.

What Makes This Research Different

1. Scale and Rigor

Most LLM security evaluations in 2026 still rely on anecdotal jailbreak attempts or narrow academic datasets. Eddie's study tested 30 models across 12 distinct offensive categories:

This is not a toy benchmark. These are 2023–2025 red team trends that real adversaries use in production engagements.

2. The "Unexpected Champions" Phenomenon

Eddie's most important finding: the models that perform best are not necessarily the ones Western enterprises trust most.

• Alibaba Tongyi DeepResearch-30B topped the leaderboard at 77.08% — demonstrating functional understanding of exploit chains, not just documentation recall.
• Mistral-7B-v0.2-Base achieved 75.00% with a perfect 100.0 in ETW_Bypass and Syscall_Shellcode — proving that smaller, efficient models can be potent force multipliers.
• Meanwhile, widely-deployed models like Llama 3.1 scored only 31.25% — not because they are "safer," but because they lack operational depth.

The defensive implication is stark: attackers are not limited to the models your organization approves. They will use whatever works best.

3. The "Script Kiddie Trap" vs. Operational Capability

Eddie correctly identifies a critical distinction:

"Numerous models generate generic code but fail to circumvent modern defenses such as EDR. They possess theoretical knowledge of exploits but lack the capability for operational implementation under defensive pressure."

This matters for defenders because not all AI-generated threats are equal. A model that outputs a generic PowerShell snippet is annoying. A model that generates a working AMSI bypass with proper P/Invoke and memory patching is a genuine escalation.

The benchmark's scoring system — 0% for ethical refusal, 50% for plausible but broken code, 100% for working, accurate output — is designed precisely to surface this distinction.

Key Takeaways for the Blue Team

Eddie's analysis translates benchmark data into actionable defensive intelligence:

"Security Through Obscurity" Is Dead

"The proficiency of models like Alibaba-NLP_Tongyi in ADCS_ESC1 (68.8%) and AMSI_Bypass (81.2%) effectively obsoletes 'Security through Obscurity'."

If you are still relying on the assumption that attackers do not understand your ADCS misconfigurations or your custom AMSI bypass signatures, that assumption is now quantifiably false.

Speed of Exploitation Approaches Zero

"The latency between CVE disclosure and weaponized script availability is approaching zero."

When a 4-bit quantized model on consumer hardware can outperform massive cloud models in shellcode generation, the barrier to entry for sophisticated attacks has collapsed.

The Arms Race Is Local

"The 2026 landscape is defined not by a singular super-intelligence, but by thousands of localized, fine-tuned, and highly capable models operating on local hardware."

This is perhaps the most important insight. Defenders must stop thinking about "ChatGPT security" and start thinking about model-agnostic threat models. Your adversary is not using the API you monitor. They are using a quantized GGUF on an air-gapped workstation.

The Final Paradox — And Why It Matters

Eddie closes with a statement that should be framed in every SOC:

"Defending against AI-generated attacks necessitates the deployment of AI-generated defenses. The cybersecurity domain is entering an era of automated warfare, where the human operator's role shifts from tactical execution to strategic command."

This is not fear-mongering. It is a measurement-driven conclusion from 30 models, 12 categories, and hundreds of test runs.

The benchmark was designed to answer one question: "Can this AI assistant actually help a red team operator in a real engagement?" Eddie's study proves that for some models, the answer is yes — which means defenders must assume the same capability is available to their adversaries.

Why This Research Deserves Attention

As the benchmark author, I have seen the framework used in various contexts — some defensive, some less so. Eddie Oz's application of it is exactly what I had in mind when building the tool:

1. Objective measurement over anecdotal claims
2. Defensive framing over capability bragging
3. Actionable conclusions over academic abstraction
4. Responsible disclosure with clear ethical boundaries

The disclaimer at the end of Eddie's article — "Using AI for offensive cyber operations without authorization is illegal" — is not boilerplate. It is a professional boundary that separates security research from criminal activity.

Conclusion

"LLMs Under Siege" is more than a benchmark report. It is a strategic assessment of where AI security stands in mid-2026:

• Capabilities are commoditized. Shellcode generation, EDR bypass, and certificate abuse are no longer niche skills.
• Model provenance does not predict risk. The "safest" Western models may be the least capable defensively.
• Local deployment changes everything. You cannot defend against what you cannot see.
• AI must augment defense, not just offense. The only sustainable response is AI-driven defensive automation.

If you are a CISO, a blue team lead, or an AI safety researcher, read Eddie's full analysis. The data is open, the methodology is transparent, and the conclusions are uncomfortable — but necessary.

References

• "LLMs Under Siege: The Red Team Reality Check of 2026" — Edilson Osorio Jr.
• toxy4ny/redteam-ai-benchmark — Benchmark framework
• OWASP LLM Top 10 — Industry risk framework
• AI Act (EU) — Regulatory context for GPAI systems

The author is a certified offensive security professional and the maintainer of the redteam-ai-benchmark open-source framework. Views expressed are personal and do not represent any employer or client.