27.5KB language-agnostic WebGPU syntax highlighter
A developer released gpu-lexer, a 27.5KB language-agnostic syntax highlighter that uses a tiny WebGPU model to label code tokens in the browser.
gpu-lexer splits source into words, whitespace, and symbols, then a small WebGPU model uses local and whole-file context to assign nine token classes, working on languages never seen in training. On held-out files, 12.57% of token labels differ from Shiki, though this measures agreement with Shiki rather than objective correctness. In benchmarks against Shiki 4.4.3, Prism.js, Highlight.js, Sugar High, and Starry Night, it highlighted 10 concatenated copies of three.min.js (5.56M characters) about 10x faster on an Apple M4 Pro in Chrome 152. The author frames it as an experiment, not a grammar-equivalent highlighter.
- Single 27.5KB bundle handles any language without grammars
- Labels plain, comment, string, keyword, type, and other token classes
- 12.57% label disagreement with Shiki on held-out files
- ~10x faster than Shiki on 5.56M-character benchmark input
Full article367 words · extracted from gpu-lexer.vercel.app · click to collapse
import { parse } from 'gpu-lexer'
const spans = await parse('source code')
// {
// type: 'plain' | 'comment' | 'string' | 'number' | 'keyword' | 'type' | 'function' | 'constant' | 'operator'
// start: number
// end: number
// }[]gpu-lexer splits source code into simple parts—words, whitespace, newlines, and symbols. Then a tiny WebGPU model combines local and whole-file context to label each part. It is designed for any language: instead of choosing a grammar, it guesses each part's type from the surrounding source, even when it never saw that language or syntax during training. Adjacent labels become the syntax spans returned to your code.
This is an experiment, not a grammar-equivalent highlighter. On files kept out of training, 11.98% of the current model's token labels differ from Shiki. This measures agreement with Shiki—not objective correctness—and unseen languages or real-world code may differ more often.
One browser run after one warm-up on September 9, 2026. The input was 10 concatenated copies of three.min.js (5.56M characters). MacBook Pro, Apple M4 Pro, 20-core GPU, 24GB, macOS 26.6.2, Chrome 152. Each engine ran in a dedicated worker; DOM rendering was excluded. gpu-lexer and Shiki returned token data, Starry Night returned a HAST tree, while Sugar High, Prism.js, and Highlight.js returned highlighted HTML. Sugar High 2.3.1, Prism.js 1.30.0, Highlight.js 11.12.0, Starry Night 3.11.0, and Shiki 4.4.3.
Minified and Brotli-compressed browser bundles measured on September 9, 2026. Major web includes javascript, typescript, css, html, json, and markdown. gpu-lexer uses the same bundle for every language. Starry Night totals include its Oniguruma WASM payload.
Shiki is the 100% normalization reference. Each library's token names are mapped to the same nine classes: plain, comment, string, number, keyword, type, function, constant, and operator. Scores compare non-whitespace source parts across 1,103 held-out files in the GitHub Innovation Graph top 25 for 2026-Q1, weighted by each language's pusher count. Unsupported languages score zero; corpus size does not affect the weights.
[ live demos ]
75 languages
react.development.js0 bytes
Text extracted automatically; images, tables and formatting may be missing. Original: https://gpu-lexer.vercel.app/