IBM Research adds consistency guidelines to ALTK-Evolve to reduce agent reliability gaps

Agentic benchmarks commonly report Mean@k, the average pass rate across k runs, which obscures variability in repeated attempts. IBM Research’s ALTK-Evolve team highlights a different metric—Pass^k, the fraction of tasks where the agent succeeds in all k runs—and shows a 24.4-point gap between Mean@5 (77.4%) and Pass^5 (53.0%) for a ReAct agent using GPT-4.1 on the AppWorld benchmark. This consistency gap indicates tasks the agent can sometimes solve and sometimes cannot, even when nothing about the task changes between runs.

To address this, the team introduces consistency guidelines, a new guideline type in ALTK-Evolve built on top of a diagnostic tool called the Consistency Analyzer. The analyzer replays each decision step in a recorded trajectory by resampling with k=5 completions per step, measuring how much the model’s output varies at that point. This requires one additional model call per decision step, performed offline against the already-recorded context, and yields a consistency score per step that flags flip-prone decisions.

The detected flip-prone steps are converted into targeted consistency guidelines in ALTK-Evolve’s standard format, which are then injected back at inference time. Applying these guidelines halves the consistency gap—from 24.4 points to 12.0 points—without degrading average accuracy. The improvements are observed both for the same task (Pass^5 +16.0 percentage points) and for similar tasks (Pass^5 +13.0 percentage points).

The team explains that flip-prone decisions arise from flat token probability distributions at critical steps, where small perturbations—such as platform-side effects or slight shifts in probabilities across runs—can flip the outcome. Greedy decoding and fixed seeds do not address the underlying distribution shape, so the problem persists even at temperature zero. Consistency guidelines directly target these flat distributions by reinforcing the most reliable decision paths identified during diagnosis.

The methodology and evaluations are detailed in a technical report on arXiv, and the consistency guidelines are integrated into ALTK-Evolve’s existing pipeline for storage and retrieval, requiring no model internals or additional instrumentation beyond the recorded trajectory.