Apple details local and distributed inference for large language models on Mac Studio with MLX and RDMA

Apple’s Machine Learning Research team describes how its open-source MLX framework enables local and distributed inference for large language models on Mac Studio with Apple silicon. The framework leverages the unified memory architecture of Apple silicon, allowing researchers and developers to run new inference and fine-tuning methods privately on their own hardware.

The team reports running Muse Spark Open, a frontier-class model with nearly a trillion parameters, on a cluster of four Mac Studios, achieving over 36 tokens per second. This performance is measured at a context size of 8,192 tokens using the latest version of MLX, released alongside the post.

Apple benchmarks three open-weight models—Muse Spark Open (816B parameters), Mistral Medium 3.5 (128B), and Qwen 3.8 (27B)—on Mac Studio configurations with M3 Ultra and M5 Ultra chips. Muse Spark Open is evaluated in 4-bit (NVFP4) precision, while Mistral Medium 3.5 and Qwen 3.8 are evaluated in their native precisions (FP8 and BF16, respectively).

The M5 Ultra chip’s GPU Neural Accelerator delivers significant speedups for compute-bound tasks: time-to-first-token (TTFT) is up to 4.6× faster on M5 Ultra compared to M3 Ultra, and generation throughput is 1.6× faster. These gains are attributed to the M5 Ultra’s 1.2 TB/s unified memory bandwidth, compared to 819 GB/s on M3 Ultra.

For distributed inference, Apple uses RDMA over Thunderbolt 5, introduced in macOS 26.2, to scale models across multiple Mac Studios. The team demonstrates two parallelization schemes: pipeline parallelization across two Mac Studios and tensor parallelization across two or four Mac Studios. Muse Spark Open, which requires more than 1.6TB of RAM in its native precision, is shown running across multiple machines using these methods.

Energy efficiency is highlighted, with peak power consumption of about 200W for Mac Studio with M3 Ultra and 300W for M5 Ultra during inference at a context size of 65,536 tokens. Apple notes that this power draw is a fraction of common household appliances, making the setup viable for home or office environments.

The MLX framework is positioned as a tool for private, local AI experimentation, enabling agentic experiences and large-context applications without cloud dependency. The team emphasizes the simplicity of implementing pipeline parallelization, requiring only minimal code changes.