AI
smolmaxxing runs 8B and 16B models on devices with about 1GB of memory
The smolmaxxing project posted on Hacker News explores running large language models on low-spec devices such as TV boxes, old tablets, single-board computers, and routers. Through custom kernels, trimmed vocabularies, and paging experts in from flash storage, the project enables 8B and 16B mixture-of-experts models to run in about 1GB of RAM.
The project advocates running the largest possible language models on minimal hardware. TV boxes, old tablets, single-board computers, and routers typically have CPUs, memory, and flash storage but sit idle most of the time. smolmaxxing measures the model size a device can actually run, then pushes the limits further through custom kernels, careful memory management, and streaming weights from flash; the figures in the repository are all measured locally on devices.
The project is based on patches to llama.cpp. Core methods include writing kernels for specific chips, such as a 1-bit lookup-table kernel for Cortex-A55, repacked 2-bit and ternary formats, and corresponding versions for the A7, which lacks dot-product instructions.
Methods for reducing memory usage also include deleting rarely used vocabulary output rows in small models; on TV devices, “headless” mode stops the launcher and Google services while the model is running. For mixture-of-experts models, the system keeps only frequently used experts in memory, reads other experts from flash on demand, and prefetches content needed for the next layer, allowing 8B and 16B models to run in about 1GB of RAM. For dense models that exceed memory capacity, it also supports streaming from flash at full flash speed.
The project says these techniques are not tied to a single device: the 1-bit kernel requires only basic NEON, and the flash-streaming approach works on Linux or Android devices where the model is larger than memory. Current examples include an ARMv8.2 Android TV box and a jailbroken iPad running iOS 12, with future support planned for phones, single-board computers, and routers. Using these models requires the project’s patched version of llama.cpp; the original version cannot load the corresponding files.