Apple study finds elaborate agent ‘harnesses' add little value over minimal coding agents for ML engineering tasks

A new paper from Apple’s Machine Learning Research group argues that elaborate agent ‘harnesses’—such as multi-agent orchestrators and dedicated retrieval subagents—do not improve performance over a minimal coding agent when evaluated under equal time budgets and using the same frontier LLM backbone.

The researchers propose that the primary driver of performance in autonomous machine learning engineering (MLE) tasks is the underlying LLM, not the complexity of the agent’s scaffolding. Their conclusion is based on large-scale systematic ablation studies comparing open-source state-of-the-art harnesses against a baseline that grants the LLM direct access to execution via read, write, and bash primitives.

The authors—Kirill Brilliantov, Alejandro Hernández-Cano, and Emmanuel Abbé—note that while modern MLE agents are often motivated by concerns over progress stagnation in long-horizon cycles and limitations of LLM primitives, the empirical evidence in this study suggests those concerns are not addressed by adding more machinery. Instead, the minimal-harness coding agent performed comparably on public leaderboards under the tested conditions.

The paper situates its findings within the broader context of agentic AI research, where increasing emphasis is placed on orchestration layers, retrieval systems, and multi-agent coordination. The authors suggest that effort spent elaborating hand-crafted harnesses may yield diminishing returns for current MLE benchmarks, and call for a re-evaluation of where engineering complexity is most effectively applied.