AWS details architecture to scale MoE reinforcement learning on Amazon EKS with EFA and DeepEP

AWS described an architecture on Amazon Elastic Kubernetes Service (Amazon EKS) that integrates Elastic Fabric Adapter (EFA) and DeepEP to scale Mixture-of-Experts (MoE) reinforcement learning workloads. The post, authored by Ashvin Nihalani, Alexander Zhipa, James Park, and Vijay Rajakumar, focuses on the infrastructure challenges of post-training MoE models with Reinforcement Learning from Human Feedback (RLHF) or Group Relative Policy Optimization (GRPO) at large scale.

The authors identify three simultaneous challenges: coordinating heterogeneous compute for rollout generation and policy training, sustaining high-throughput communication across hundreds of accelerators, and dynamically orchestrating subsystems to maintain balance. They propose using Amazon EKS for orchestration, EFA for high-bandwidth, low-latency networking, and DeepEP as the communication library to address these issues.

The post reports that this combined architecture increased aggregate reinforcement learning rollout throughput by 40% for large-scale RLHF and GRPO training, attributing the gain to improved communication efficiency and coordinated resource scheduling.

MoE has become a standard architecture for scaling large language models to hundreds of billions or even trillions of parameters by leveraging sparsity for efficient inference. However, the authors note that sparsity does not eliminate infrastructure complexity during training, particularly in the reinforcement learning stage, which requires elastic inference alongside tightly coupled model training.