Multi-objective reinforcement learning method improves Pareto-optimal RTL circuit generation with LLMs

Researchers propose a multi-objective reinforcement learning method that optimizes large language models (LLMs) for generating Pareto-optimal RTL circuits from natural-language specifications. Unlike prior approaches that aggregate power, performance, and area (PPA) objectives into a single scalar reward, the new method explicitly targets the diversity and quality of the entire set of generated circuits across the PPA objective space.

The proposed technique consists of two components: a generation-based training pipeline that reframes coverage improvement of the Pareto solution set as a sequential circuit generation problem, and a reward function based on Hypervolume Contribution (HVC) that quantifies how much a newly generated circuit improves the overall Pareto front.

In evaluations on the RTLLM v2 benchmark, the method achieves statistically significant gains: a +13.3 percentage point improvement in relative Hypervolume over a baseline without PPA optimization, and a +14.4 percentage point improvement over a single-objective reinforcement learning approach. The authors also report superiority in circuit diversity, indicating that the method produces a broader and more useful range of trade-off solutions.

The authors note that in practical hardware design, the relative importance of PPA objectives varies by product requirements and constraints, making a diverse Pareto set valuable for downstream selection. They position this work as the first to optimize, not a single circuit’s quality, but the quality and diversity of the entire candidate set generated across multiple runs.

The paper has been accepted at COLM 2026, indicating peer-reviewed recognition within the machine learning community.