Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules
Cortex keeps coding-agent skills and rules only when replayed tests show a net gain.
Cortex converts developer corrections of coding agents into replayable tasks and keeps a proposed skill or rule only when repeated runs show a net gain and no task becomes noticeably worse. It neither adds nor deletes text the agent never used. With Claude Haiku 4.5 on a purpose-built repository, the resulting harness improved convention-following on held-out tasks, mainly through automatically loaded rules. Under Claude Sonnet 5, and on a rarely corrected open-source project, Cortex found little or nothing to learn. The authors release the tool, lab, and data.
- Harness edits are kept only after repeated with-and-without replay tests.
- Auto-loaded rules produced most gains; always-on skills were rarely opened.
- Cortex retained none of the planted unused suggestions.
- Claude Sonnet 5 and a real project yielded little new text.
Full article334 words · extracted from huggingface.co · click to collapse
Coding agents such as Claude Code also follow text files that developers write for them: a project file, skills the agent may choose to open, and rules that load automatically when the agent reads matching files. We call these files the agent's harness. Developers add to it whenever they correct the agent, but one success proves little: the same agent can pass a task once and fail it the next time. We present Cortex, a tool that tests changes to the harness instead of trusting them. It saves fixes from the developer's sessions as replayable tasks, finds the corrections that recur, writes one skill or rule at a time, and keeps it only if repeated runs with and without it show a net gain and no task made noticeably worse. Existing text is tested the same way before it is deleted, and Cortex neither adds nor deletes text the agent never used. We tested Cortex with Claude Haiku 4.5 on a repository built for the purpose, with conventions the model tends to break, one area with nothing to learn, and a script playing the developer. On held-out tasks, the harnesses it built made the agent follow the conventions far more often than no skills or rules did, left the comparison area unchanged, and added no text read on every turn. Rules, which load automatically, did most of the work; skills helped only when offered with the files a task touches, and those offered on every turn were rarely opened. Cortex kept none of the useless or harmful suggestions we planted, but the agent never opened the useless ones or the harmful skills, and it reached no verdict on the harmful rules, so the test shows that Cortex keeps no unused text, not that it detects damage. Under the stronger Claude Sonnet 5 a single run's harness added little, and on a real open-source project, where the agent was rarely corrected, Cortex found nothing to learn. We release the tool, the lab and its data.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.14216