Study finds hosted System-1 decision model outperforms open-weight counterpart on agent harness tasks

A new arXiv preprint evaluates two System-1 decision models—Laya (open-weight) and Jev (hosted)—on 11 agent decision points derived from 18 public sources, comprising 7,283 base cases and 6,640 robustness variants. The evaluation used byte-identical inputs, paired tests, and reproducibility checks across hardware and days.

Jev significantly outperformed Laya on 9 of the 11 decision points, with accuracy gains ranging from +10.8 to +46.0 percentage points. Neither model surpassed chance-level performance on zero-shot model routing, and both models tied on RAG relevance gating.

The study also examined robustness properties. Laya changed 30% of its answers when option order was reversed and degraded sharply as the number of similar candidate tools increased, achieving only 31% accuracy at 50 nearest-neighbour tools compared to Jev’s 98% on items with a unique correct tool.

In a self-audit, the authors identified four errors and confounds in their initial deployment claims. These included an omitted pre-screen cost that inflated reported savings (23.9% vs. actual 4.3%), gate accuracy being reported as end-to-end quality (58% vs. 98%), in-sample thresholds leading to up to 17% held-out misses, and a 'channel effect' on injection false positives that disappeared when using channel-native content.

The authors released all cases, raw outputs, and analysis code in a public repository, enabling independent verification of the results and the identified corrections.