Qwen3-0.6B mixes 4-bit decode with selective 8-bit steps
An entropy gate at a 16% escalation rate scores 62.7% on 150 GSM8K problems, using 0.87 of all-high bytes.
Stories per day · 30 days
9 RSS
Relatedgguf11huggingface7inference6qwen6deepseek5kv-cache5embeddings3fp83gemma3linear-attention3mixture-of-experts3open-weights3privacy3reinforcement-learning3comfyui2federated-learning2fp42language-models2
135 stories, newest first · one card per story
Stories tagged #quantization
Sable proves bit-exact quantized LLM inference on one GPU
Qwen2.5-32B takes 92.5 seconds for 4,096 tokens; proofs are not zero-knowledge.
BAQ-TD keeps TD learning rates over rate-limited channels
Sending O(d log T) bits over horizon T is enough for the order-optimal 1/T rate, up to logarithmic factors.
Paper introduces e-means, a faster mini-batch k-means variant
Authors report 10-45x GPU speedups and a no-sweep rule for decay rate and batch size.
FP4 pretraining needs more exposures to learn rare facts
NVFP4 needed 1.32× and 1.52× as many exposures as BF16 to match recall while loss differed by 0.5% and 2%.
mradermacher publishes GGUF quants of gemma-4-31B-it-ARA-heretic
Static GGUF files from Q2_K through Q8_0, plus mmproj supplements, are listed as the repo trends at #30.
Paper measures mismatch on-policy distillation can tolerate
Model-specific FP8 recipes kept quality and fit at least 1.93x as many KV-cache tokens as BF16.
BitDecoding-FP4 enables accurate FP4 KV cache decoding
Algorithm-system co-design achieves near-BF16 accuracy with 2.4-3x higher throughput for LLM serving.
arXiv benchmark finds int8 quantization nearly matches float32 on text embeddings
int8 quantization retains 94.4% baseline effectiveness at 32x storage reduction across 38 MTEB tasks.
Video Rebirth publishes HyperFlow H3 Flash, a compressed MiniMax-H3
The 13.7B–13.9B model makes 5.2–15 second video with stereo audio in 4–8 steps, under a restricted MiniMax license.
LoopVQ sub-1-bit KV cache quantization for looped transformers
LoopVQ jointly quantizes channels across multiple loop steps in looped transformers to reduce memory while preserving accuracy.
PlantVLM-Edge enables quantized vision-language models for on-device crop diagnosis
4-bit quantization retains 98.6% answer quality while reducing memory by 49%, running in 0.12s for first token on edge devices.
Scantist/GLM-5.3-SLIM-GGUF 180GB MoE model trends on Hugging Face
179.9GB version of 753B-parameter GLM-5.3 MoE drops to 563B by NAS, scores 92.7% HumanEval and fits single B200.
1
FLAMIA kernels accelerate attention on NVIDIA Blackwell GPUs
FLAMIA achieves 1.10-1.40x speedup for long-context attention using MXFP8 on B200 GPUs with token-block scaling.
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-i1-GGUF trending on Hugging Face
GGUF quantized uncensored DeepSeek model trending at #30 on Hugging Face.
Correlated quantization cuts communication in private federated RL
One-bit uploads with correlated private quantization reduce mean travel time by 14.3% while preserving privacy in vehicular networks.
Study compares human and Mixtral-simulated clients in counselling role-play
Novice counsellors practiced with peers and with 4-bit Mixtral 8x7b clients; sentence-level patterns both matched and diverged.
Gradient-aware allocation and sliding windows for low-bit GDN states
On Qwen3.5-4B and 9B, GGA plus PW-S8 beats STEPQuant@4 by 38.39 and 41.21 points on seven-task averages.
DeepSeek-V4.1-Flash-1017 Uncensored Model Trending on Hugging Face
AtomicChat releases 763B DeepSeek-V4.1-Flash-1017 uncensored GGUF quantized for single 12GB GPU with Strata.
1
WISD self-distills reasoning models using a deliberately weaker copy
Pruning raises Qwen3.5-4B from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6.
4-8 bit quantized DNNs stay exposed to membership inference
Eight attacks across six quantization methods find privacy gains mainly at 2 bits, with large utility loss.
Secure aggregation scheme supports heterogeneous client quantization in federated learning
It avoids leaking partial aggregates when clients quantize differently and up to T semi-honest clients collude with the server.
ThinkBit preserves reasoning in scalar two-bit language models
On Qwen3-1.7B, a 0.52 GB recalibrated checkpoint scores 46.7 on MATH-500 versus 30.0 for EfficientQAT.
Guide covers running Underdog Saluki 27B for tool calling
The title describes Saluki 27B as a 2-bit Qwen3.8-27B build; the page body was not captured.
1