Hugging Face publishes dataset and Space to check AWS Trainium compatibility for Transformer models

Hugging Face introduced a Space and a public dataset to help users determine whether a Hugging Face Transformer model runs unmodified on AWS Trainium using the torch-neuronx backend.

The Space, titled 'torch-neuronx model support,' displays results for a curated selection of popular models based on downloads and trending scores. It indicates whether a model runs unmodified on Trainium, the number of Trainium cores it is sharded over, the wall time of the run (including compilation), and the peak device memory.

Users can search for specific models by name or browse by task using the Space’s Trending or Downloads views, which align with Hub rankings. If a model was excluded, the Space explains why, such as lack of prioritization or incompatibility.

The Space also provides the Neuron stack version, required packages, and a run.py script to execute the model on a Trainium2 (trn2) instance. The script loads the model with from_pretrained, shards it when necessary, and generates text using a static cache and torch.compile(backend='neuron').

For models that require sharding, the Space offers a command to download pre-compiled graphs from the public NEFF cache bucket, avoiding recompilation during subsequent runs.

Hugging Face defines 'runs' as loading the model on a trn2 instance sharded over the smallest number of Trainium cores that can hold its weights, generating 32 tokens, and validating the output. On a single core, token outputs are compared with a CPU run of the same checkpoint; for sharded models, a small judge model checks for fluent English output.

The results are sourced from the public dataset aws-neuron/torch-neuronx-model-support, which contains one Parquet file per pair of neuronx-cc and library versions. Each row includes a model’s result, error details if it failed, stack versions, and the run.py script used.

At the time of writing, the tool supports only Transformer models and is limited to text-generation and image-text-to-text tasks, with plans to expand to diffusers, sentence transformers, and other Transformer tasks in the future.