harshatheg/Qwen-2.5-1B-RLCD — new model trending #30 on Hugging Face
A community MLX inference engine evaluates constrained JSON schema fields in parallel on Apple Silicon, reporting 5.6-7.0x latency speedups with guaranteed schema validity.
The repository harshatheg/Qwen-2.5-1B-RLCD appeared at #30 on Hugging Face trending, but its content describes Parallel Constrained Decoding, an MLX-based inference engine for structured extraction and classification on Apple Silicon Macs. Benchmarked with mlx-community/Qwen2.5-1.5B-Instruct-4bit on an M4 Max, it reports 5.6x-7.0x latency reductions (e.g., 1,900 ms to 270 ms for a 28-field support triage task) with 100% syntactic validity and calibrated field-level probabilities. The engine prefills a single KV-cache, broadcasts it across all schema fields, and slices logits to valid candidate tokens for enum fields with up to 255 choices.
- KV-cache broadcasting evaluates all schema fields simultaneously instead of token-by-token autoregressive generation.
- Reported 5.6x-7.0x speedups on M4 Max, e.g., 270 ms versus 1,900 ms for a 28-field triage schema.
- Sub-vocabulary logit slicing over candidate sets yields 100% valid JSON with calibrated softmax confidence scores.
- Requires Apple Silicon (M1-M4) and macOS 14+; enum fields support up to 255 categorical choices.
- Organizations
- Hugging Face
- AI models
- Qwen2.5-1.5B-Instruct-4bit
From the original article
6 min read at huggingface.cotext-generation · 0 downloads · 145 likes · license apache-2. 0
By harshatheg · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.