Jin Daily AI Trivia - Flash-MoE squeezes Apple Silicon to the limit
Flash-MoE is an extreme demonstration of truly understanding the Apple Silicon memory architecture. By using a precise “use only what you need” I/O approach, it breaks the long-standing assumption that the entire model must be loaded into RAM to run.
Flash Weight Streaming for MLX (Flash MoE write their own metal engine) allows models larger than your RAM to run on Apple Silicon — 30B on 16GB, 70B+ on 32GB+, with no additional quantization and full native precision.
This implementation is inspired by Apple Research’s LLM in a Flash (arXiv:2312.11514).
And here’s the crazy part: Running Qwen3.5 397B-A17B on my own 48GB M4 Pro hits around 4.5 tokens/sec.
Check out the guy GitHub on how much crazy optimization he did on Apple Silicon memory architecture to get this feat done.
