Research
CWE-bench v2: Open security agent benchmark
Hugging Face Bloghigh confidence
Why it matters
- →Provides a standardized, reproducible way to measure AI agent reliability in critical security tasks.
- →Reveals that even frontier models fail nearly 20% of complex vulnerability fixes, setting a realistic performance baseline.
- →Open-source infrastructure allows immediate community evaluation and integration into defensive agent training pipelines.
Collinear AI has released CWE-bench v2, an open-source benchmark designed to evaluate AI agents' ability to identify and fix security vulnerabilities in real open-source repositories. The dataset contains 100 tasks spanning 91 different Common Weakness Enumerations (CWEs), with deterministic programmatic scoring to ensure reproducibility. Initial evaluations show Gemini 4 Argon leading at 67.8% pass@1, followed closely by Claude Opus 5.5, though both models fail approximately 19% of tasks on every attempt. The benchmark is fully open, including the dataset, verifiers, and a Docker-based execution environment called Harbor, allowing developers to test their own models against these security challenges.
Read originalThe story around this
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
How we got here
- 1OpenAI Questions Reliability of SWE-Bench Pro
OpenAI · July 8, 2026 · Related
- 2AI Agents Involved in Uncontrolled Hacking Incidents
WIRED AI · August 4, 2026 · Background
- 3OpenAI reveals PHASEONE agent swarm security incident
Wes Roth · August 27, 2026 · Background
- 4New AI Safety Benchmarks: Agents Last Exam and Terminal Bench
AI Explained · September 4, 2026 · Related
- 5AISI releases verified evals for GPT-5 and Claude Opus 4
Hugging Face Blog · September 22, 2026 · Background
- 6Nvidia Launches Open-Source AI Agent Security Platform
WIRED AI · September 28, 2026 · Background
- 7ThinkingBox reveals agent reliability gap
Hugging Face Blog · October 3, 2026 · Related
- 8AI agents lack real-time observability standards
MIT Technology Review AI · October 29, 2026 · Background
What happened next
- 9Major AI security incidents and vulnerabilities
Fireship · November 3, 2026 · Background
- 10BAAI Releases AREX-2 Agent Model
Sam Witteveen · November 6, 2026 · Background
- 11Hundreds of Artex AI hacking tools exposed
TechCrunch AI · November 6, 2026 · Background
More from Hugging Face Blog
© Hugging Face BlogResearchagents
ToyBlocks: Controlled simulation for robot learning research
Hugging Face launched ToyBlocks, a controlled simulation framework designed to isolate and study the specific mechanics of robot learning. By focusing on a simple tabletop environment with blocks, it allows researchers to automatically generate massive datasets of tasks and demonstrations without manual teleoperation. This setup enables precise testing of how high-level reasoning models like Molmo 2 interact with low-level action controllers like ManiFlow. The real value lies in reproducibility; scientists can now systematically vary training data to understand exactly why a robot fails or succeeds, moving beyond the black-box nature of current robotics benchmarks.
Hugging Face Blog
© Hugging Face BlogResearchother
Open ASR Leaderboard adds Egyptian and Nigerian English
The Open ASR Leaderboard is expanding its evaluation scope by adding private test sets for Egyptian and Nigerian English, addressing a significant gap in global representation. This update moves beyond the traditional Inner Circle accents to include Outer and Expanding Circle varieties, providing a more realistic benchmark for speech recognition models. By partnering with Liva AI for data collection, Hugging Face ensures these evaluations reflect natural, high-context user interactions rather than artificial tasks. This shift forces developers to optimize for diversity, making model performance metrics more trustworthy and globally relevant.
Hugging Face Blog
© Hugging Face BlogModels & Labsvideo
NVIDIA Cosmos 3 Open Models for World Simulation
NVIDIA is opening up its Cosmos 3 world model family with four new 16B parameter checkpoints, shifting from closed research to open-weight simulation. The release targets two critical domains: autonomous driving and robotics. For self-driving cars, the models generate synchronized camera video and LiDAR data, solving the alignment problem between visual appearance and 3D geometry. For robotics, Sim-Bimanual predicts environmental changes from robot actions, enabling closed-loop policy training. This moves world modeling from passive video generation to interactive simulation where agents can act and observe consequences.
Hugging Face Blog
More in Research
© The AI Daily BriefResearchresearch
AI-Led Research Acceleration Reported by Meter
Meter reports that research driven by artificial intelligence is accelerating, highlighting a trend in scientific and technical discovery.
The AI Daily Brief
© MIT News AIResearchother
MIT's CheMeleon predicts molecule behavior from scant data
CheMeleon is a foundation model for chemistry that learns general molecular patterns from a million PubChem entries rather than specific property labels. This pre-training allows it to outperform classical expert-driven models and other AI baselines when fine-tuned on just a few data points, solving the critical bottleneck of sparse experimental data in drug discovery. Its impact is already visible: nine of the top twenty entries in a major OpenADMET contest used it, and one pharmaceutical company has made it their default starting point for modeling. This shifts molecular screening from trial-and-error to pattern-based prediction, significantly reducing the need for expensive synthesis.
MIT News AI
© WIRED AIResearchother
Open Chinese AI Models Match Frontier Labs in Hacking
The long-held assumption that closed frontier models hold a definitive edge in cyber capabilities has collapsed. Vals research shows DeepSeek matching OpenAI’s Daybreak on complex network attacks, completing 73% of tasks versus 81%. This parity arrives just months after Anthropic predicted such a shift, rendering gated access strategies largely moot for slowing capability proliferation. The cost disparity is stark: $2 to $3 for open models versus up to $46 for proprietary APIs, creating a powerful economic incentive for widespread adoption by both defenders and attackers. We are now in a landscape where advanced offensive AI is cheap, accessible, and effectively democratized.
WIRED AI