harshatheg/Qwen-2.5-1B-RLCD — new model trending #30 on Hugging Face
A community MLX inference engine evaluates constrained JSON schema fields in parallel on Apple Silicon, reporting 5.6-7.0x latency speedups with guaranteed schema validity.
The repository harshatheg/Qwen-2.5-1B-RLCD appeared at #30 on Hugging Face trending, but its content describes Parallel Constrained Decoding, an MLX-based inference engine for structured extraction and classification on Apple Silicon Macs. Benchmarked with mlx-community/Qwen2.5-1.5B-Instruct-4bit on an M4 Max, it reports 5.6x-7.0x latency reductions (e.g., 1,900 ms to 270 ms for a 28-field support triage task) with 100% syntactic validity and calibrated field-level probabilities. The engine prefills a single KV-cache, broadcasts it across all schema fields, and slices logits to valid candidate tokens for enum fields with up to 255 choices.
- KV-cache broadcasting evaluates all schema fields simultaneously instead of token-by-token autoregressive generation.
- Reported 5.6x-7.0x speedups on M4 Max, e.g., 270 ms versus 1,900 ms for a 28-field triage schema.
- Sub-vocabulary logit slicing over candidate sets yields 100% valid JSON with calibrated softmax confidence scores.
- Requires Apple Silicon (M1-M4) and macOS 14+; enum fields support up to 255 categorical choices.
Full article1,355 words · extracted from huggingface.co · click to collapse
# Parallel Constrained Decoding for Apple Silicon
[](https://huggingface.co/spaces/drinkmoonshine/parallel-constrained-decoding)
> **Live Demo**: Try the side-by-side comparison live on Hugging Face Spaces: [drinkmoonshine/parallel-constrained-decoding](https://huggingface.co/spaces/drinkmoonshine/parallel-constrained-decoding).
A high-throughput inference engine for structured information extraction, decision routing, and categorical classification on Apple Silicon using MLX.
Parallel Constrained Decoding evaluates multi-field JSON schemas simultaneously rather than generating tokens sequentially. On an Apple Silicon M4 Max, it delivers **5.6x to 7.0x latency reductions** compared to standard autoregressive decoding with **100% schema validity** and **calibrated field-level confidence scores**.
---
## Performance Benchmarks (Apple Silicon M4 Max)
Evaluated with `mlx-community/Qwen2.5-1.5B-Instruct-4bit` on macOS Sequoia:
| Scenario | Fields | Autoregressive Baseline | Parallel Constrained | Latency Speedup | Syntax Validity |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Fintech Fraud Routing** | 4 fields | 420 ms (120 tok/s) | **75 ms** | **5.6x** | 100% guaranteed |
| **Code Security Audit** | 4 fields | 380 ms (125 tok/s) | **68 ms** | **5.6x** | 100% guaranteed |
| **High-Cardinality Tariff** | 1 field (255 choices) | 500 ms (118 tok/s) | **89 ms** | **5.6x** | 100% guaranteed |
| **Enterprise Support Triage** | 28 fields | 1,900 ms (130 tok/s) | **270 ms** | **7.0x** | 100% guaranteed |
---
## Why Parallel Constrained Decoding?
### The Problem with Autoregressive Structured Generation
Standard LLM structured generation (such as JSON mode or grammar-guided sampling) relies on token-by-token autoregressive decoding:
```
[Context Prompt] -> "{" -> "\n" -> " " -> "risk" -> ":" -> " " -> "HIGH" -> ...
(Requires 150 to 500 sequential forward passes)
```
Each token requires a distinct GPU/NPU forward pass and sequential memory bandwidth roundtrips. As schema size grows, latency scales linearly with output token length:
$$T_{\text{autoregressive}} = \sum_{k=1}^{K} t_{\text{step}}(k)$$
Additionally, autoregressive decoding is susceptible to syntax degradation, field omission, and hallucinated keys.
### The Solution: Parallel Evaluation via KV-Cache Broadcasting
In structured extraction and classification, field values belong to bounded candidate sets (booleans or categorical enums). Parallel Constrained Decoding exploits this property:
```
+---> [Field 1: "risk_level"] -------> Logit Slicing -> Top Choice
|
[Context Prefix Prefill] -+---> [Field 2: "requires_review"] ---> Logit Slicing -> Top Choice
(Single KV-Cache State) |
+---> [Field M: "action_tier"] ------> Logit Slicing -> Top Choice
(All fields evaluated simultaneously)
```
1. **Single Broadcast Prefill**: The context document and semantic schema descriptions are prefilled once into an MLX Key-Value (KV) cache.
2. **KV-Cache Broadcasting**: The KV-cache is broadcast across all $M$ schema fields in parallel.
3. **Sub-Vocabulary Logit Slicing**: For each field, only candidate token IDs belonging to valid schema choices are evaluated. The remaining vocabulary is masked.
4. **Calibrated Softmax Probabilities**: Exact normalized probabilities are calculated over the candidate slice:
$$P(c_i) = \frac{\exp(z_i / T)}{\sum_{j=1}^{C} \exp(z_j / T)}$$
5. **Token Tree Disambiguation**: When candidate choices share multi-token prefix roots, the engine executes continuation steps using sliced cache states with zero memory reallocation.
6. **Programmatic Assembly**: Output JSON is constructed directly from verified values, guaranteeing 100% valid syntax without JSON parsing errors.
---
## Installation
### Prerequisites
- Apple Silicon Mac (M1, M2, M3, M4 series)
- macOS 14.0 or later
- Python 3.10+
### Setup
Clone the repository and install dependencies:
```bash
git clone https://github.com/your-org/parallel-constrained-decoding.git
cd parallel-constrained-decoding
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```
---
## Developer SDK Quickstart
### 1. Defining Schemas
Schemas are defined using `StructuredSchema`. Each field specifies a `type` (`enum` or `boolean`), a `description` to guide model reasoning, and `choices` (for enum types, supporting up to 255 choices):
```python
from core.schema import StructuredSchema, FieldDefinition
# Option A: Dictionary-based definition
schema_dict = {
"priority": {
"type": "enum",
"choices": ["P0_CRITICAL", "P1_HIGH", "P2_NORMAL", "P3_LOW"],
"description": "Urgency tier based on customer business impact"
},
"requires_escalation": {
"type": "boolean",
"description": "Whether an on-call engineer must be notified immediately"
},
"department": {
"type": "enum",
"choices": ["BILLING", "INFRASTRUCTURE", "SECURITY", "PRODUCT_SUPPORT"],
"description": "Target handling department"
}
}
schema = StructuredSchema(schema_dict)
```
You can also construct fields explicitly using `FieldDefinition`:
```python
fields = {
"tariff_classification": FieldDefinition(
name="tariff_classification",
field_type="enum",
description="Harmonized System 6-digit tariff category code",
choices=["0101.21", "0101.29", "8471.30", "8517.12", "8542.31", ...] # Up to 255 choices
)
}
```
### 2. Running Parallel Generation
Execute parallel constrained inference on your context string:
```python
from core.engine import run_parallel_generation
context = """
Incident Report: Production database db-primary-01 CPU at 100%.
Payment gateway failing for 40% of checkout requests.
Tier 1 Enterprise customer affected: Acme Global.
"""
result = run_parallel_generation(context, schema)
print(f"Latency: {result['elapsed_ms']} ms")
print(f"Prefill Time: {result['prefill_ms']} ms")
print(f"Passes: {result['sequential_forward_passes']}")
print("\nExtracted JSON:")
print(result["parsed_json"])
```
### 3. Response Structure
The output dictionary provides both the structured JSON and detailed field telemetry:
```python
{
"mode": "parallel_constrained_calibrated",
"elapsed_ms": 74.5,
"prefill_ms": 52.1,
"suffix_eval_ms": 18.2,
"sequential_forward_passes": 1,
"is_valid_json": True,
"schema_match": True,
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/harshatheg/Qwen-2.5-1B-RLCD