[object Object]
Huntress raised Claude Fable 5.1 Rails API recall from 48% to 100% with three harness changes.
Huntress tested Claude Fable 5.1 on Rails API recall with a fork of the rails/lemans harness and seven tasks taken from real application code. With no guidance, the agent used the built-in Rails feature in 10 of 21 trials (48%), near the public Agents on Rails score of 41% for Fable 5.1; cited leaderboard scores were 34.9% for Opus 5 and 25.4% for Sonnet 5. Familiar APIs such as generates_token_for scored 83%, while perform_all_later, comparison validators, and has_secure_token scored 0% because the agent copied existing hand-rolled helpers. One CLAUDE.md rule raised recall to 86%, and adding a gem lookup tool plus a second model to review the diff reached 100% on the seven tasks.
- Unguided baseline used built-in Rails APIs in 10 of 21 trials.
- One CLAUDE.md rule alone raised accuracy from 48% to 86%.
- Gem lookup and a second-model diff check reached 100%.
- Agents copied hand-rolled helpers already present in the codebase.
- Vendors
- Anthropic
- Products
- Ruby on Rails
From the original article
11 min read at huntress.comTL;DR: When Huntress tested Anthropic's Claude Fable 5. 1 in evaluations, we started with a ~48% accuracy baseline for API recall: significantly more than previous models but still pretty low. With three small changes to our coding harnesses, we got it to 100% accuracy across seven of our eval tasks.
huntress.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.