Skip to content
Agents · Nov 3, 2026

New benchmark finds self-evolving agents struggle to learn hidden rules from feedback alone

CLR-Bench evaluates six methods; only a stronger model with specific techniques surpasses a simple history baseline.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • A new benchmark, CLR-Bench, tests whether self-evolving agents can learn hidden rules from iterative feedback.
  • Six recent self-evolution methods showed minimal gains over a frozen agent when evaluated with a certifying model.
  • A stronger model (GPT-6-Sol) paired with ACE and Dynamic Cheatsheet methods reached 78% accuracy using feedback alone.
  • The benchmark includes 20 chains across 10 domains, each adding one hidden rule per level.

Researchers introduce CLR-Bench, a benchmark designed to test whether self-evolving agents can learn rules that are not explicitly provided. The benchmark presents agents with a curriculum where each level adds one hidden rule on top of previous ones, operating over opaque labels representing named groupings published online, such as airline alliance founding members.

Each task provides one example value, practice feedback reveals one additional member and one non-member, and the test evaluates performance on unseen members and similar non-members. Probes confirm that the base model knows the members of each grouping but does not infer the grouping without feedback, and that evidence correctly identifies the grouping once labels are interpreted as named groupings.

The authors evaluate six recent self-evolution methods using a certifying model and find minimal learning: the best method gains at most 5 points over a frozen agent and excludes only 6–12% of unseen members, which is less than the 20% achieved by simply retaining raw practice history. A fixed lesson that explicitly states the named-grouping convention improves performance by 14.5 to 16.5 points for the same methods.

The study reports that agents’ persistent states retain revealed members in 75–98% of cases but almost never capture the class name. When practice exposes a missing member, agents add that member to their list rather than inferring the underlying rule. With a stronger model (GPT-6-Sol) and specific techniques (ACE and Dynamic Cheatsheet), agents reach 78% accuracy and solve 59–60% of single cells, outperforming both the certifying model baseline (53% and 8%) and the raw history approach.

The benchmark was built using an automated pipeline to construct 20 chains spanning 10 domains, providing a reusable evaluation framework for future work on self-evolving agents.

Sources
  1. 01arXiv cs.AI — More Rules, Longer Lists? Benchmarking Self-Evolving Agents on a Hidden-Rule Curriculum
Also on Agents

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.