The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Edge0 streams 35B-parameter MoE inference from SSD on a 24GB machine at 20 tokens/s using a one-token-ahead prerouter, with framework and checkpoints open-sourced.