Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills
Anthropic ships a plugin evals workflow for Claude Code with six grader types, a no-plugin baseline arm, and a CI gate via threshold and cost flags.
Anthropic published a plugin evals workflow for Claude Code, exposed via the "claude plugin eval" command on v2.1.269+. Six grader types exist: regex, tool_used, tool_order, and file_exists are free transcript checks, while llm and baseline invoke a billed judge model. Every case runs with and without the plugin, and the delta (Δ) isolates the plugin's contribution; a Δ near zero with a failing tool_used:Skill grader indicates the skill never triggers. CI gating uses --threshold 0.8, --max-cost-usd, --trust-plugin, and --no-publish flags, with results written to a report.html under evals/results/.
- Six grader types: regex, tool_used, tool_order, file_exists, llm, baseline; the first four cost nothing.
- Every case runs with and without the plugin; delta measures what the plugin contributed.
- CI gate flags include --threshold, --max-cost-usd, --trust-plugin, and --no-publish.
- claude plugin eval init proposes cases and graders automatically.
- A near-zero delta with failing tool_used:Skill means the skill does not trigger on natural phrasing.
- Vendors
- Anthropic
- Products
- Claude Code
- Organizations
- Anthropic
From the original article
3 min read at marktechpost.com
Anthropic has published a new plugin evals workflow for Claude Code. The claude plugin eval command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run where the plugin is not loaded. It answers 3 questions plugin developers could not previously measure: does the skill trigger, does it […]
By Asif Razzaq · marktechpost.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.