Hugging Face details AsyncGRPOTrainer with LoRA support for distributed RL training across Jobs

Hugging Face announced that AsyncGRPOTrainer in TRL v1.14 now supports training a LoRA adapter and syncing only that adapter to vLLM replicas, rather than the full model. A rank-1 adapter for a 1.5B model is a few megabytes, enabling efficient transfer compared to gigabytes for the full model.

The adapter sync path leverages Hugging Face Storage Buckets mounted as volumes in each Job, eliminating the need for NCCL or a shared filesystem. The trainer writes the adapter to a shared bucket path, and vLLM replicas read from the same path to load the latest adapter version.

A small proxy server routes each rollout to the vLLM replica most likely to hold the relevant KV prefix and broadcasts every adapter update to all replicas. The proxy communicates with the trainer over localhost and with replicas over HTTPS.

The setup consists of three Jobs: one AsyncGRPOTrainer Job with LoRA and FSDP, two vLLM Jobs serving the base model plus the latest adapter, and a Storage Bucket mounted at the same path in all three Jobs. The bucket also persists checkpoints and the final adapter, allowing a preempted trainer to resume training.

Each vLLM replica reserves adapter slots based on max_staleness in AsyncGRPOTrainer. With max_staleness=4, up to five policy versions may be in use at once, requiring six adapter slots during swaps. The replicas use the stock vllm/vllm-openai image with runtime LoRA loading enabled.

Five experimental runs using the same recipe showed training time for 500 steps reduced from 3 hours 27 minutes to 53 minutes. The runs varied by microbatch packing and checkpointing strategies, illustrating where bottlenecks occur in the pipeline.

LoRA is particularly suited for RL because the advantage function provides limited information per episode, making a low-rank adapter sufficient. Prior work from Thinking Machines showed that rank-1 adapters can match full fine-tuning in policy-gradient RL.

The AsyncGRPOTrainer already separates training and generation, but this setup extends it to run on separate machines without shared local disk or NCCL by using Storage Buckets as a shared filesystem substitute.