Research
ThinkingBox reveals agent reliability gap
Hugging Face Bloghigh confidence
Why it matters
- →Agents can pass tool-call validation while leaving databases in invalid states.
- →Single-shot success rates are poor predictors of production reliability.
- →Terminal state verification is essential for evaluating autonomous agents.
Microsoft and Hugging Face released ThinkingBox, a benchmark evaluating AI agents on their ability to maintain correct backend state across 507 business workflows. The study found that 67% of execution failures involved agents completing tool calls correctly but leaving the database in an invalid state. While Kimi-K3 demonstrated higher initial task coverage, Claude Opus 5.5 proved superior in consistency, maintaining performance across 20 repeated trials. The benchmark highlights a significant reliability gap between successful API calls and actual operational outcomes.
Read originalThe story around this
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
How we got here
- 1Benchmarking Open Models for Agentic Use
Hugging Face Blog · June 18, 2026 · Related
- 2MIT and Microsoft Enhance AI Workflow Efficiency
MIT News AI · June 25, 2026 · Related
- 3Agentic AI Gains Confidence in Tech Workflows
MIT Technology Review AI · June 29, 2026 · Related
- 4OpenAI Questions Reliability of SWE-Bench Pro
OpenAI · July 8, 2026 · Related
- 5Enterprise AI Faces Evaluation Trust Gap
VentureBeat AI · July 16, 2026 · Related
- 6New AI Safety Benchmarks: Agents Last Exam and Terminal Bench
AI Explained · September 4, 2026 · Related
- 7Enterprise AI Agent Pilots Rarely Reach Deployment
AI News · September 14, 2026 · Related
- 8Hugging Face Tackles AI Consistency with New Tool
Hugging Face Blog · September 15, 2026 · Related
More from Hugging Face Blog
© Hugging Face BlogOpen Sourcewriting
Ai2 open-sources AstaBrief 8B for scientific reports
Allen Institute for AI has released AstaBrief 8B, an open-weight model designed specifically for generating cited scientific literature reviews. Built on Qwen3-8B and trained with supervised fine-tuning and direct preference optimization, it prioritizes speed and grounding over complex multi-step reasoning. The model generates full reports in a single pass, cutting generation time to roughly 51 seconds compared to the 178 seconds required by proprietary alternatives like Claude. This release offers researchers a faster, locally deployable option for synthesizing evidence without relying on external APIs.
Hugging Face Blog
© Hugging Face BlogAgentsagents
ServiceNow AutoSynthData automates agent training data
ServiceNow CoreAI released AutoSynthData, a pipeline that turns enterprise agent failures into targeted training datasets. By using a stronger teacher model to identify capability gaps and generate feasible, realistic tasks, it solves the bottleneck of creating high-quality synthetic data for specific environments. The system validates every generated task against strict verifiers before adding it to the curriculum, ensuring the model learns from actual weaknesses rather than noise. This approach shifts agent training from manual curation to automated, continuous improvement loops grounded in real-world constraints.
Hugging Face Blog
© Hugging Face BlogResearchmodels
Hugging Face Olmo-core 3 scales MoE training to trillions of parameters
Hugging Face’s Olmo-core 3 rewrites the rules for open-source Mixture-of-Experts training by shifting from FSDP to DDP, keeping experts resident on GPUs to slash communication overhead. This architectural pivot yields a 2.7x throughput jump on NVIDIA B300s and enables stable training of models with over one trillion total parameters while keeping active compute fixed. By integrating MXFP8 precision and optimized routing, the framework closes the efficiency gap with proprietary stacks like Megatron-Core. Researchers now have an open, battle-tested infrastructure to build massive sparse models without relying on closed-source enterprise tools.
Hugging Face Blog
More in Research
© MIT News AIResearchresearch
MIT research solves RL sensitivity in transportation
Cathy Wu’s team at MIT has cracked a persistent bottleneck in reinforcement learning: its notorious sensitivity to specific problem setups. By identifying that RL models train effectively on only about 10 percent of related problems, they developed an algorithm to select those high-yield training cases. This approach boosts training efficiency by up to 30 times, allowing researchers to generalize solutions across complex transportation networks without retraining from scratch. The method transforms RL from a fragile proof-of-concept into a viable tool for evidence-based policy design, specifically showing eco-driving could cut emissions by 11-22 percent.
MIT News AI
© MIT Technology Review AIResearchresearch
Why LLMs Don't Actually Reason
A former Google DeepMind researcher argues that current LLMs lack genuine reasoning capabilities, relying instead on fast pattern matching rather than the deliberative search mechanisms seen in AlphaGo. The core issue is that LLMs maintain no persistent, inspectable epistemic state, meaning they cannot track hypotheses or evidence systematically. This architectural flaw makes them unreliable for high-stakes fields like medicine and science where auditability is critical. True machine intelligence requires a separation between knowledge representation and manipulation, moving beyond next-token prediction to auditable inference.
MIT Technology Review AI
© AI ExplainedResearchresearch
OpenAI Publishes Research on AI-Driven Intelligence Explosions
OpenAI has released a new research paper exploring the potential for AI systems to recursively improve themselves, leading to rapid intelligence growth.
AI Explained