Apple introduces REVERSAL-BENCH to study reset-free reinforcement learning in irreversible environments

Apple’s Machine Learning Research group introduced REVERSAL-BENCH, a benchmark designed to study reset-free reinforcement learning (RL) in environments where actions cannot be undone. The work highlights a core challenge in autonomous RL: many existing paradigms assume environmental reversibility, a property absent in real-world manipulation tasks such as pushing objects off tables or spilling granular substances.

The benchmark introduces two key components: a continuous reversibility parameter ρ∈[0,1] that controls the degree of reversibility in the environment, and a reset oracle—a ground-truth mechanism to verify whether a given state is recoverable. These tools enable researchers to test state recoverability across eight manipulation settings implemented in five physics engines.

In evaluations spanning a broad spectrum of policy architectures—including standard actor-critic algorithms, safe RL, and specialized reset-free frameworks—the team observed a sharp 'reversibility cliff.' As ρ increases, reset-free agents consistently enter irrecoverable states from which they cannot recover, halting further learning. In contrast, episodic agents maintained steady learning performance under the same conditions.

The authors attribute this failure mode to irreversibility rather than obstacle complexity, confirming it through comparisons with geometrically identical reversible counterparts. The breakdown persists even in full physics simulations under learned manipulation policies, underscoring the practical challenge of deploying reset-free agents in irreversible settings.

Alongside the benchmark, Apple released a large multi-simulator dataset labeled with recoverability labels and a reset oracle. The team also evaluated a safety shield designed to intervene before irreversible failures occur, finding that while recoverability can be predicted accurately, active recovery primarily succeeds only when the agent can physically steer clear of the trap.