NaiveAI ships open-weights Naive-N0.5-Flash for fast coding
A small Latent Space test saw high server-side throughput and very long hard-question traces, without grading accuracy.
1
Stories per day · 30 days
16 RSS
Relatedopen-weights22agents7flint5multimodal5pretraining5benchmarks4deepseek4language-models3minimax3model-release3nemotron3quantization3agentic-coding2benchmark2coding-agents2coding-models2gguf2inference2
132 stories, newest first · one card per story
Stories tagged #mixture-of-experts
InfraRSI harness applies recursive self-improvement to LLM training
On four dense and MoE workloads, the authors report up to 2.75x speedup and mean MFU rising from 17.4% to 30.0%.
MoE Routing Strategies: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing Explained
The guide covers seven production MoE routing methods used in models like Mixtral and DeepSeek-V4, highlighting sigmoid gating and bias balancing as dominant approaches.
1
Thinking Machines Inkling 2 Small claims top spot among Western open-weight models
Inkling 2 Small scores 31 on Artificial Analysis Intelligence Index, 18 points above predecessor, at higher cost.
1
onóna uses twin tied embeddings to cut validation loss
At matched token budgets it lowered loss in 11 of 12 runs and raised average accuracy in 10 of 12.
Anthropic open-sources FlashProve provable inference prototype
FlashProve cryptographic proofs confirm specific models generated specific replies, tested on Llama 4 Maverick MoE model.
1
UniRestorer++ scales image restoration with degradation experts
The paper reports roughly log-linear PSNR gains and a 50-category CDD-50 benchmark.
MAEP prunes MoE multimodal experts by modality in one pass
The training-free method cuts about 27 GB of BF16 expert weights so an about 58 GB model fits a 40 GB GPU.
CATF framework improves context-aware time-series forecasting
Manager-Worker architecture with hard routing outperforms standalone models and Time-MoE on five context-rich datasets.
DepthGraft shares KV hierarchically to regularize deep LLMs
On a 16B MoE model the authors report +7.0 points on MMLU-Pro while cutting KV cache by about one third.
Samaya releases Sterling, a 3B-active finance research model
On Samaya’s own benchmark it scores 58.9% at $0.61 per question versus 55.9% and $6.89 for Claude Fable 5.1.
Prime Intellect ships trainable fused Mega MoE kernel for faster training
Self-reported full steps on Qwen3 MoE models were 32% to 34% faster than prime-rl and up to 55% faster than TorchTitan.
2
Multi-View MoE reuses multiple routers to shape expert computation
On a 24-layer 10B-A1B model trained on 708B tokens, nine-task mean rose 1.18 points.
Calibrated distillation consolidates specialists in dynamic MoE models
On Turing-20B-A2B the method scores 0.9988 across four domains and beats Qwen3-8B on math and code.
NVIDIA releases open Nemotron 3.5 Transcribe, a 30B MoE speech model
The open model transcribes 40 languages while activating about 4 billion of its 30 billion parameters.
AIR schedule targets effective learning rate, not nominal rate
On 128 billion-token MoE pretraining, AIR lowers terminal training loss by about 0.01 versus cosine at matched cumulative learning rate.
HotPOT pairs PDE-specific numerical experts in one operator transformer
Across 24 matched comparisons, HotPOT cuts rollout error 50.7% on average versus MoE-POT, including a 17M model.
MuonGS relaxes weight-norm control for LLM pretraining
On dense models it reports about 15% token-equivalent speedup versus MuonH at matched validation loss.
Muon optimizer doubles first MoE layer bias span
Muon raises router bias span in early MoE layers, complicating load balancing at tuned learning rates.
MoE channel selection works without calibration data, paper finds
A weight-only reader keeps 98.1% of full accuracy at 17.8% of routed-expert reads and speeds MoE blocks versus vLLM.
Moonshot launches Kimi K3.1 Pro; weights promised November 4
The 2.8T mixture-of-experts model, with 104B active parameters, scored 55 on Artificial Analysis; the API is live but weights are not.
DiffuSpace releases Clover-35B-A3B-Preview diffusion language model
The text-only model uses about 3B active parameters per token and a 128K context window.
1
Moonshot AI releases open-weight Kimi K3.1 Pro
The title lists 2.8T total parameters, 104B active parameters, and a 1 million-token context.
2
LLM code self-refinement often raises energy versus human solutions
Of 44 configurations, 42 used more energy per problem, with overheads from 4% to 886%.