Hugging Face publishes open pipeline to train a coding model to paint watercolors via reinforcement learning

Hugging Face published an end-to-end, open-source reinforcement learning pipeline that trains a coding model to paint watercolors by generating JavaScript code using the p5.brush library. The pipeline uses TRL and OpenEnv, with all artifacts—training scripts, RL environment, scorer model, and trained models—published on the Hugging Face Hub.

The project reproduces a prior art project by Surya Narreddi, who demonstrated watercolor paintings generated by a language model writing JavaScript. The open implementation includes a reference pool dataset, an RL environment, training scripts, and trained models, all available in a dedicated Hugging Face collection. The pipeline can be launched with a single command using Hugging Face Jobs, Spaces for the RL environment and scorer model, and Inference Providers for the pairwise judge.

The reward function combines four terms: a gate to ensure the sketch compiles and renders without cheating, a length penalty, a pairwise judge comparing the candidate painting to four randomly selected references from the pool, and HPSv3, an open 7B preference model that scores aesthetic preference. The pairwise judge is Qwen3-VL-30B-A3B-Instruct, called via Hugging Face Inference Providers.

Three training runs were conducted with different weight splits between the pairwise judge and HPSv3: judge-led (0.60/0.30), hps-led (0.30/0.60), and hps-only (0.00/0.90). The runs were stopped at steps 110, 110, and 60 respectively. The project’s goal is to explore reinforcement learning over aesthetic preference, where correctness is not verifiable and the reward is shaped by curated taste encoded in the pool and models.

The author notes the project’s focus on reproducing the original idea with open artifacts, including a hand-rated pool and three reward mixes for comparison. All code, models, and training configurations are published, and the pipeline runs entirely on Hugging Face infrastructure.