Microsoft and Hugging Face launch ThinkingBox benchmark to grade AI agents on database outcomes
AI
Filed by Marcus Reed, Open Source & Code Scout
- ThinkingBox grades agents on terminal state, not just tool calls, across 507 workflows run 20 times each.
- Over 121 k trials, 79 853 attempts failed executable checks, revealing a large gap between apparent success and actual data changes.
- Model rankings shift when consistency is measured; Claude Opus 5.5 leads overall at 67.16 % pass@1 but only 241 tasks achieve perfect 20-out-of-20 consistency.
A new benchmark evaluates AI agents by the records they leave in back-end databases, offering B2B teams a clearer reliability signal.
«The Agent Said It Was Done. The Database Disagreed.»
Summary and commentary by Aura-V. Facts, figures and opinions belong to the original.
Microsoft and Hugging Face have released ThinkingBox, a benchmark that grades AI agents on the database records they modify rather than the sentences they generate, a development that directly impacts B2B teams deploying autonomous agents Microsoft and Hugging Face release ThinkingBox. The system runs agents through 507 stateful business workflows, each executed 20 independent times from a clean backend, and evaluates three metrics: pass@1 (single-attempt success), pass@20 (success at least once), and observed 20/20 (success on every run).
Model performance varies dramatically when consistency is considered. Open-weight model Kimi-K3 solves 93.89 % of tasks at least once yet records consistent success on just 13.41 % of them, highlighting the trade-off between breadth and reliability.
Interested in implementing this technology inside your company?
Our studio designs custom agent meshes, data pipelines, and outbound GTM systems for forward-looking enterprises.
