Hugging Face details PRX model's data pipeline in Part 4 of technical series

Hugging Face published the fourth part of its PRX technical series, focusing on the data strategy underpinning the PRX model. The post frames pre-training as a problem of coverage and diversity, arguing that broad, representative corpora teach models more about the structure of the visual world than smaller, highly curated sets.

The team assembled a large, diverse dataset from a mix of public and internal sources, prioritizing breadth and leveraging existing curation rather than rebuilding it at scale. They used a vision-language model (VLM) to re-caption images, aiming for long, accurate captions that describe all visible elements, including screenshots, advertisements, and text in images. This approach treats such elements as controllable attributes rather than noise to be removed.

For data formats, the team combined Lance for feature engineering and dataset curation with Mosaic Data Shards (MDS) for streaming during distributed training. Lance's columnar format and predicate pushdown enabled efficient dataset building and exploration, while MDS allowed for mixing, shuffling, and distributed training directly from object storage.

The post also details compute and quality trade-offs. Switching the text encoder from T5Gemma to Qwen3-VL during training incurred only a roughly 3–4% throughput cost, avoiding the need to pre-compute and store text latents and enabling future encoder changes without rewriting terabytes of data.

On image encoding, the team measured the impact of JPEG compression at quality 92 versus lossless PNG. Repeated encode/decode cycles showed JPEG quality 92 to be essentially imperceptible after the first re-encode, with minimal further degradation. Storing images as JPEG at quality 92 reduced storage footprint by 3–10× compared to PNG with no perceptual loss, given that most source images were already JPEG-compressed.

The pipeline emphasizes pragmatic, scalable engineering over absolute optimality, positioning the dataset as a solid starting point for pre-training a 7B model rather than a final, polished artifact.