Hugging Face Storage integrates with SkyPilot for zero-egress AI workloads across clouds

SkyPilot, a multi-cloud orchestration tool, now supports Hugging Face Storage as a first-class backend via the hf:// scheme, allowing users to mount Hugging Face Buckets or Hub repositories directly into AI workloads. The integration enables compute jobs to read and write data without incurring cross-cloud egress fees, addressing a common pain point for teams running GPU workloads across multiple cloud providers.

With this change, SkyPilot tasks can reference Hugging Face storage using a single hf:// URL and an existing HF_TOKEN, eliminating the need to manage per-cloud storage accounts or duplicate datasets across regions. The hf:// scheme supports both MOUNT and COPY modes, with MOUNT enabling lazy loading of files so that only the data accessed by the job is fetched on demand. This reduces startup time and keeps GPUs productive from the outset, particularly during the first epoch of training when nothing is cached.

Hugging Face Storage charges no egress or CDN fees, so compute jobs launched by SkyPilot on AWS, GCP, Azure, or other supported clouds can read data directly from the same bucket without additional transfer costs. The storage backend is built on Xet, which deduplicates incremental checkpoints and model variants, further reducing storage and transfer overhead. Authentication is handled via the HF_TOKEN environment variable, which SkyPilot passes securely to the job, ensuring consistent access across clouds without managing separate credentials for each vendor.

The joint effort between Hugging Face and SkyPilot includes upstream contributions to the hf-mount FUSE backend, enabling unprivileged container support. This allows the Hugging Face Storage mount to behave like other SkyPilot storage backends, such as S3 or GCS, while preserving the zero-egress benefit. The integration is designed to fit into existing workflows: most teams already store models and datasets on the Hugging Face Hub, so no migration is required to start using SkyPilot for multi-cloud GPU workloads.