28.9M-parameter language model runs on an $8 microcontroller without server support

A GitHub project demonstrates running a 28.9 million parameter language model entirely on an ESP32-S3 microcontroller that costs about $8, generating text at roughly 9.5 tokens per second without sending data to a server. The model stores 25 million of its parameters in the device’s 16MB flash and uses only 512KB of SRAM for the active compute core, relying on Google’s Per-Layer Embeddings technique to sample rows from flash as needed.

The project reports end-to-end throughput of about 9.5 tokens/sec and pure compute throughput of 9.7 tokens/sec, with the model quantized to 4-bit and totaling 14.9MB in size. Most of the embedding table resides in slow flash and only about 450 bytes are pulled into SRAM per token, keeping the working set small while the bulk of the model remains resident in non-volatile storage.

The model was trained on the TinyStories dataset, which consists of short synthetic stories designed to be learnable by small models, so it generates simple, mostly coherent stories but cannot answer questions, follow instructions, or write code. The authors note that these limitations stem from the small reasoning core rather than the memory layout trick, which is primarily an architectural demonstration.

All artifacts are open-source: firmware and wiring instructions live under firmware/esp32_llm/README.md, training and quantization code under src/ and experiments/, and detailed on-chip measurements, ablations, and method write-ups under RESULTS.md. The repository also documents a correction from an earlier inflated parameter count after a bug in accounting was fixed.