Hidden Reasoning: A New Cost‑Efficient AI Benchmark

AI & Machine LearningCognitive EnhancementRobotics

Imagine an AI that learns a new visual rule from a few examples and solves it instantly, all without speaking its thoughts aloud.

TL;DR
• BDH‑CQ combines in‑context learning with latent reasoning, reaching 29.5% pass@2 on ARC‑AGI‑1 at $0.0007 per task.
• Controlled experiments show strong contextual binding, extrapolation, and composition limits.
• The system sets a new cost‑accuracy frontier, opening doors for scalable, efficient reasoning models.

Summary
BDH-CQ introduces a hybrid reasoning system that merges in‑context learning with recurrent latent computation, allowing demonstrations to update a persistent memory and queries to be solved through iterative high‑dimensional transformations without verbalizing intermediate steps. The model, built on a 150 M‑parameter BDH architecture, was trained on a curated mix of ARC‑style datasets and evaluated on the public ARC‑AGI‑1 benchmark, achieving 29.5% pass@2 at a computed cost of $0.00070 per task—well below the prior cost‑accuracy frontier.
• Deploy models that update recurrent memory from demonstrations to reduce token overhead and inference latency.
• Prioritize training on diverse, task‑structured data mixtures to improve generalization across visual transformation families.
• Incorporate controlled demonstration coverage to mitigate extrapolation failures in depth‑sensitive operations.
• Monitor consistency metrics (strict‑task vs. test‑pair accuracy) to detect partial rule application before large‑scale rollout.
• Scale reasoning effort incrementally, balancing higher pass@2 gains against marginal cost increases.

The evaluation on ARC‑AGI‑1 shows a pass@2 of 118/400 tasks (29.5%) with an average compute of 0.85 H200 GPU‑seconds per task, translating to less than one‑tenth of a cent. Independent black‑box audits reproduced these results and extended analysis to ConceptARC, where performance varied across 16 concept families—high scores in FilledNotFilled and TopBottom2D (9/10) contrasted with low scores in Copy and Order (2/10). Test‑pair accuracy often exceeded strict‑task accuracy, revealing that the system sometimes generates correct outputs for individual inputs without consistently applying the inferred rule.

Controlled experiments further mapped capability boundaries. Propagation and copying operations extrapolated reliably up to the tested limits, while ordering and nesting showed sharp declines beyond length‑eight and depth‑five respectively. Providing a single supporting demonstration at the target complexity restored a substantial portion of performance, indicating that demonstration coverage directly influences extrapolation success. Composition tests demonstrated robust handling of rotation with relocation (72/72) but weaker performance for reflection‑relocation (47/72) and color‑swap compositions (0/72), highlighting representation‑dependent limits.

Scaling analyses revealed that increasing latent reasoning effort improves pass@2 (HIGH 29.5%, MEDIUM 27%, LOW 21%) with modest cost reductions. The authors propose extending BDH‑CQ to larger model sizes, longer training regimes, and broader domains such as language and mathematics, while retaining the same memory‑reasoning fabric. Future work will also explore integrating verbalized chains of thought when external communication or verification is required, ensuring that continuous latent reasoning remains the primary engine for efficient, adaptable AI systems.

How can we harness latent reasoning to build AI that learns on the fly while staying affordable?