Hugging Face publishes profiling guide for PyTorch attention implementations

Hugging Face published the third installment in its "Profiling in PyTorch" series, focusing on attention mechanisms and how to analyze them with PyTorch’s profiler. The post builds on earlier parts that profiled basic math operations and fused linear layers, now turning to attention as the next fundamental algorithm to optimize.

The authors walk through a naive attention implementation composed of primitive PyTorch operations—matrix multiplications, scaling, causal masking, softmax, and another matrix multiplication—then profile it to reveal the sequence of kernels executed on CPU and GPU. They show how an out-of-place masked_fill introduces an unexpected memory-copy kernel, which inflates latency.

To address this, they switch to an in-place masked_fill_ operation and rerun the profiler. The updated trace removes the memory-copy kernel on the GPU lane, demonstrating a one-line change that reduces overhead per attention forward pass. The savings are small for a single operation but can accumulate across many layers in large models such as LLMs and diffusion models.

The post also highlights that in-place operations save time and memory by avoiding extra tensor copies, which is beneficial when working with large tensors like attention logits. It cautions that in-place operations are only safe under torch.no_grad, as they would otherwise corrupt gradient computation during backpropagation.

Finally, the guide points to PyTorch’s built-in scaled dot-product attention (SDPA) as an optimized alternative that encapsulates the entire pipeline into a single function, encouraging readers to compare its profiler trace against the custom implementations.