MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes
MetroLLM-Bench is a 955-case benchmark testing language models as transit kiosk tool-calling runtimes across six real metro systems.
The benchmark covers 37-414-station metro systems and eleven task categories including routing, fare calculation, disruptions, accessibility, and adversarial input, with 14 deterministic and 8 semantic scoring components. Of 26 models from six vendors, a PEFT-tuned 4B Qwen 3.5 student scored 91.3 on Tier 1, exceeding GPT-5.6 (90.6/90.0), while Muse Glimmer 30B led the composite ranking. A deterministic rule-based baseline reached 84.6, and PEFT gains over base models shrank from +7.03 points at 2B to -0.91 at 27B.
- PEFT-tuned 4B Qwen 3.5 beats GPT-5.6 on Tier 1 (91.3 vs 90.6)
- Fits in a 2.6 GB Q4_K_M footprint for kiosk deployment
- Serving configuration alone shifts Qwen comparisons by 2.7 Tier 1 points
- Rule-based baseline scores 84.6; LLM advantage is in compound and temporal scenarios
- Benchmark, harness, and fine-tuned students released on GitHub
- Products
- MetroLLM-Bench
From the original article
1 min read at huggingface.co
We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing, fare calculation, disruptions, accessibility, and adversarial input.
By Remco Hendriks · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.