Running a 28.9M parameter LLM on a microcontroller

GitHub – slvDev/esp32-ai · GitHub

This is a 28.9 million parameter language model that generates text on an ESP32-S3 microcontroller. It runs on the chip itself, with nothing sent to a server, and it displays generated text at 9.88 tokens per second on a small screen wired to the chip. It fits because most of the model lives in flash instead of RAM, using Per-Layer Embeddings, an idea from Google’s Gemma 3n.

Thanks to Michael Stauffer