AI agent tooling under the microscope: RTK cuts reported tokens but not real costs, and ByteDance Seed's HarnessDev finds LLM-built harnesses trail human engineering

Quesma's $1,500 benchmark found RTK's reported 89% token savings translate to only about 5% cost change either way on Terminal-Bench 2.1, while ByteDance Seed's new HarnessDev benchmark shows the best LLM-written agent harness averaged 67.8 versus an 86.2…

Two independent evaluations published September 11, 2026 challenge claims about AI coding-agent tooling. Quesma benchmarked RTK (Rust Token Killer), a terminal-output filter for AI coding agents with 79k GitHub stars whose README claims up to 90% output reduction, spending over $1,500 across 1,740 Terminal-Bench 2.1 attempts pairing Claude Code with Fable 5.0 and OpenCode with DeepSeek V4 Pro 0813. Although RTK's own 'rtk gain' metric reported 349.2 million tokens saved (an 89% reduction) across 445 DeepSeek attempts, total costs moved only about -5% for Fable and +5% for DeepSeek, with pass rates dropping 1-2% — a gap Quesma attributes to rtk gain counting removed bytes rather than billed tokens or money saved, and to cached terminal-output reads costing as little as 1/10 to 1/30 of regular input tokens. A bug in rtk find 0.45.0 also drove one agent into 339 consecutive errors, costing roughly 9x the baseline attempt, though the task still passed. Separately, researchers from ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI introduced HarnessDev, a benchmark that grades the runnable agent harness an LLM writes rather than its answers, using Creation and Evolution stages across 2,207 instances spanning SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, and BrowseComp. Among six creator models (Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, Seed 2.0 Pro), Opus 4.8 posted the best average of 67.8 versus an 86.2 human-engineered reference; self-built harnesses beat references on writing and ML experimentation but lag badly on code and search, and quality proved executor-specific — Opus 4.8 fell from 69.3 to 33.0 on SWE-bench Pro under a Gemini executor. Evolution gains were small and noisy: of 64 adjacent changes, feedback and held-out scores agreed only 34 times (53.1%), just 2 of 9 declared final versions were held-out optimal, and much generated machinery was inert (18 of 108 code components never fired; 124 of 587 writing features were dead code). Both studies converge on the same theme: agent-harness tooling claims — whether token-savings figures or self-generated harness quality — often fail to hold up under direct measurement.

• RTK (Rust Token Killer) is a terminal-output filter for AI coding agents with 79k GitHub stars; its README claims up to 90% output reduction.
• Quesma's benchmark spent over $1,500 on tokens across 1,740 Terminal-Bench 2.1 attempts, running Claude Code with Fable 5.0 and OpenCode with DeepSeek V4 Pro 0813.
• With RTK enabled, total costs moved only about -5% for Fable and +5% for DeepSeek, while pass rates dropped 1-2%.
• RTK's rtk gain metric reported 349.2 million tokens saved (an 89% reduction) across 445 DeepSeek attempts, but it counts removed bytes, not billed tokens or money saved — a source-claim-versus-measurement disagreement at the heart of the…
• Cached terminal-output reads cost as little as 1/10 to 1/30 of regular input tokens.
• A bug in rtk find 0.45.0 caused one agent to loop with 339 consecutive errors, costing roughly 9x the baseline attempt; the task still passed.
• Read, Grep, and Glob tool calls bypass RTK's shell rewriting.
• HarnessDev, from ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI, grades the runnable agent harness an LLM writes rather than its answers, using Creation and Evolution stages across 2,207 instances on SWE-bench Pro,…