Researchers propose Regularized Emphatic TD learning to address stability issues in off-policy reinforcement learning

Researchers have introduced Regularized Emphatic Temporal-Difference Learning (RETD), a new algorithm designed to address stability limitations in off-policy reinforcement learning under constant stepsizes. The work focuses on Emphatic Temporal-Difference Learning (ETD), which stabilizes expected off-policy TD updates and alters projection geometry, but does not guarantee stability in constant-stepsize sampled dynamics.

The authors construct an ergodic two-state counterexample demonstrating that ETD’s mean map can contract while the sampled product exhibits a positive top Lyapunov exponent, indicating instability. They separate this instability from the infinite variance of the follow-on trace using regenerative-cycle analysis. RETD is proposed as a normalized first-order post-shock repair that preserves the trace and importance ratios of ETD while storing the emphatic TD signal in a leaky scalar state and releasing a delayed correction.

RETD’s raw equilibrium is shown to be an affine shift of the ETD equilibrium, and single- and two-regularization readouts recover the ETD fixed point exactly. The authors prove almost-sure convergence for harmonic diminishing stepsizes and provide a conditional constant-stepsize moment-contraction result derived from a Markovian random-product bound.

Theoretical guarantees are complemented by empirical validation: paired 10,000-run experiments validate the separations between mean-map contraction and sampled dynamics, fixed-point recovery, a nonmonotone stability region, and task dependence. RETD changes post-shock dynamics but does not reduce the shared follow-on-trace variance, distinguishing its mechanism from prior approaches.