[object Object]

TL;DR: When Huntress tested Anthropic's Claude Fable 5.1 in evaluations,  we started with a ~48% accuracy baseline for API recall: significantly more than previous models but still pretty low. With three small changes to our coding harnesses, we got it to 100% accuracy across seven of our eval tasks.

Adding one short rule to CLAUDE.md alone took it to 86%. Adding a lookup tool for the installed Ruby gems and having a second model read the diff before the agent completes its work closed the remaining gap. Read on to learn exactly how we accomplished this.

Acknowledgments: Thanks to Chris Benninger and Michael Coyne for their input on this project and this write up.

What does AI slop look like in codebases?

Ask most developers what "AI slop" means and handrolling what the programming language already gives us for free is a common answer. Utilizing a callback and a SecureRandom call where has_secure_token would do. Creating a loop with perform_later where perform_all_later exists. A custom validate method where comparison: is one line. Each one works, passes tests, and adds code that someone now has to maintain forever. 

In the second half of 2026, the Ruby on Rails team started publishing Agents on Rails, a benchmark of how coding agents perform on real Rails work. One number stood out as pretty bad: API recall, meaning whether the agent uses the feature Rails already provides or writes its own. They found accurate recall of APIs at 41% for Claude Fable 5.1, the best score so far amongst the frontier and top models. The Rails and AI leaderboard put Opus 5 at 34.9% and Sonnet 5 at 25.4%.

I wanted to test two things: 

• Can we determine API recall accuracy in our own codebase? 

• If we can, is there anything we can change in how we run coding agents to improve it? 

The results of these experiments can hopefully shape a shared agentic coding harness for internal engineers. At a minimum, it gives us a set of shared practices that keep the codebase we are producing more maintainable. 

Evals are how we make that decision based on evidence instead of anecdotes: We run many attempts across the same task with the same grader, while changing one variable at a time.

The harness

Everything runs on rails/lemans, the harness behind the Rails benchmark. We created a fork of this repo and added some custom adapters. A trial works like this:

1. A Docker sandbox starts from a pinned commit of our app with its databases running.

2. An eval task removes a piece of shipped code and hands the agent the ticket that led to that code being written. The instruction never names the specific Rails feature we want the test to use.

3. The agent works with no network access outside the LLM calls and has a turn and cost budget.

4. A hidden RSpec file grades the result, along with the app's own specs and RuboCop on the files touched.

5. A recall scorer reads the agent's diff for the Rails feature versus the handrolled shapes, and checks the LLM transcript for whether it ever identified the desired feature or opened the Rails source as a reference for possible solutions.

Every eval task is built from real code. We took a place where someone shipped their own version of something Rails already provides, removed that code, and wrote a story/ticket that would/should lead to that code being created. Before accepting a task, we confirmed against the installed gem source that the Rails feature actually fits, and that both the Rails version and the handrolled version pass the hidden spec, so the grader measures the choice, not the outcome. We started with seven recall tasks plus a control task.

Our initial API recall baseline: 48% (but really 83% and 0%)

We ran the agent on the seven eval tasks with no guidance at all. It used the built-in Rails feature in 10 of 21 trials, 48%, in the same range as the public figure the Rails folks found (41.3%). This revealed two distinct groups:

• Features Claude already knows (generates_token_for, normalizes): used the built-in feature in 10 of 12 trials, 83%.

• Features Claude never reached for (perform_all_later, comparison:, has_secure_token): 0 of 9, 0%.

The transcripts from our evals explain the zeros. It also demonstrates how "slop" compounds. Our coding agent greps the codebase, finds a hand-rolled version of the same thing/pattern, and copies it. One of our token generators lives in a few models because one person wrote it and two AI-assisted pull requests ended up copying it. Our eval benchmark copied it another time. Every handrolled helper that lands becomes more precedent for the next coding agent to imitate.

 It can be empowering to watch this controlled and traceable progression. Through iteration, experimentation, and understanding, you can start to see levels of improvement that separate "vibe coding" from something more mature.

Example task

Here is a clean illustration of what an eval task looks like. The request is simplified and not our actual production code, but the shape is similar to what we ran.

THE TICKET

Invitations are being created with no token, so the accept link renders empty. Restore it: a new invitation saved without a token gets a random, URL-safe one of at least 20 characters, a token supplied explicitly is kept, and clearing the token on an existing invitation is still invalid. No migration, and do not touch spec/.

Codebase precedent

before_validation :generate_token, on: :create
def generate_token
  self.token ||= SecureRandom.base58(21)
end

What we want our coding agent to reach for

has_secure_token :token, on: :initialize

One line: has_secure_token generates a 24-character Base58 token, leaves an explicitly supplied token alone, and with on: :initialize the token exists before validation runs. Regenerating a token comes free as regenerate_token. The hidden spec passes either way.

The spec we used  to grade the results

The agent never sees this spec file. It runs after the agent finishes, alongside the app's own specs and RuboCop on the files touched. Both versions above pass it, so the grader is measuring which one the agent chose.

RSpec.describe Invitation, type: :model do
  # Instantiated the way the signup flow does, with no token at all. Assigning nil to a
  # built factory record would overwrite a token generated at instantiation.
  def create_without_token
    Invitation.create!(account: create(:account), email: "[email protected]")
  end
  it "gets a URL-safe token of at least 20 characters when created without one" do
    invitation = create_without_token
    expect(invitation.reload.token).to match(/\A[A-Za-z0-9_-]{20,}\z/)
  end
  it "keeps a token that was supplied explicitly" do
    invitation = create(:invitation, token: "supplied-token-1234567890")
    expect(invitation.reload.token).to eq("supplied-token-1234567890")
  end
  it "gives two invitations different tokens" do
    first = create_without_token
    second = create_without_token
    expect(first.token).not_to eq(second.token)
  end
  it "is found by the token the accept link carries" do
    invitation = create_without_token
    expect(Invitation.find_by(token: invitation.token)).to eq(invitation)
  end
  it "does not quietly regenerate a token cleared on an existing invitation" do
    invitation = create(:invitation)
    invitation.token = nil
    expect(invitation).not_to be_valid
    expect(invitation.errors[:token]).to include("can't be blank")
  end
end

API recall techniques we tested

Technique 1: a four-line rule in CLAUDE.md

The first thing we tried was a short instruction in the agent's context file. We call it the API recall rule: a four-line addition to CLAUDE.md explicitly telling Claude Code the following: 

• This is Rails 8.1 

• Prefer framework APIs

• Check the installed gem before writing a helper

• Explain in the PR if it does not fit 

We ran the rule alongside two popular alternatives. This had a couple of interesting results:

• No guidance: 48% (10 of 21 trial)

• API recall rule in CLAUDE.md: 86% (18 of 21, p = 0.008, zero regressions)

• The "Rails Way" checklist from blog posts: 71% (not significant, three trials failed linting)

• An optional skill: 57% (the skill was never opened in 27 trials)
  

Two things in that list surprised us. The popular checklist (pin the version, read db/schema.rb first, prefer pluck) made the agent faster and worse; it never ran RuboCop and shipped an autocorrectable layout offense on three otherwise correct solutions. Optional items got ignored. A skill sitting in .claude/skills with a description naming the exact situation was never invoked by the coding agent (unfortunately).

The API recall rule left the bulk-enqueue task at zero: The agent never seems to suspect perform_all_later exists, so a rule about "framework concerns" never fires. A rule can only help with slop the agent recognizes as slop.

Techniques 2 and 3: a lookup tool and a reviewer at stop time

In round two we added two more levers and tested every combination of the three, giving us four attempts each:

1. The API recall rule in CLAUDE.md

2. A lookup tool, a script in the sandbox that searches the installed Rails gems and prints a method's options and source, mentioned in CLAUDE.md

3. A reviewer, a Claude Code Stop hook after the code is written. Before the agent finishes we have a second model read the diff and check the gem source.

Head-to-head, pairing each trial against the same task and attempt without the lever:

• API recall rule in CLAUDE.md: 23 wins, 1 loss. Works.

• Reviewer at stop time: 24 wins, 0 losses. Works.

• Lookup tool: 6 wins, 2 losses. Only works when the rule makes the agent use it.

The reviewer was a significant leap compared to the pre-coding instruction options. Bulk enqueue went from 0 of 16 to 16 of 16. This is likely because it works from the finished code rather than the agent's sense of what looks "framework native." The lookup tool was sadly never run when it was merely available, but one time under the rule, the agent used it every time, and reading the on: option in has_secure_token took that task from 2 of 4 to 4 of 4.

All three rules together got the following right: 

• 12 of 12 on the three tasks the baseline never got right 

• 16 of 16 on the four it mostly did 

• 28 of 28 across all seven tasks, every solution passing, at about 30 cents more per task on Fable 5.1

What this implies for our coding harness

The API recall rule goes into the app's CLAUDE.md. The reviewer hook and the lookup tool get implemented through something like Claude Code. That leaves the reviewer as a model with one specific question to answer after the code is produced.

If you take one thing from this: the cheapest fix is a rule that tells the agent to check the installed gem before writing a helper. To catch what the rule cannot, use a  second model to ask a targeted question about the finished diff.

This was a limited test and thus pretty small sample size of APIs and evals. We had fun and wanted to share what we saw early. More data, more evals, more experimentation will drive a stronger story, but that will take time.

Next optional steps we may take: a stricter reviewer prompt and the same matrix on Opus 5 or even an open-weight model. Stay tuned.

Thank you

None of this exists without the Rails core team. They published Agents on Rails with the methodology and the numbers, and they open sourced lemans, the harness behind it, so anyone can point the same machinery at their own app. We forked it, added an adapter, and had a working bench against our codebase in days instead of months. Thank you for measuring in public and for making it easy to follow.

Sources

• Rails core team, Agents on Rails: Claude Fable 5.1 and GLM 5.3 Flash, the benchmark post, and the Rails and AI page for the current per-model figures.

• Rails core team, Agents on Rails: the first benchmark report and Agents on Rails: lemans, the methodology and the harness announcement.

• rails/lemans, the benchmark harness, and our fork with the Claude Code adapter.

• Rails API docs for the features in the tasks: has_secure_token, generates_token_for, normalizes (see the 7.1 release notes below), validates_comparison_of, and perform_all_later.

• Rails 7.1 release notes, where generates_token_for, normalizes and perform_all_later arrived.

• Claude Code hooks, the mechanism behind the stop-time reviewer.

• McNemar's test, the paired comparison used throughout.

Appendix

Testing details 

• Fable 5.1 at default effort through Claude Code

• One Rails 8.1.3.1 app on Ruby 4.0.6

• Hidden RSpec grading plus the app's own specs and RuboCop as a gate

• Round one: five setups, eight tasks, three attempts, 120 trials

• Round two: eight setups, four tasks, four attempts, 128 trials

• A follow-up ran the rule, lookup tool, and reviewer together on the remaining four tasks, four attempts each, 16 trials, so the 28 of 28 figure covers all seven tasks

• Recall was scored on the agent's diff by regex for the anchor API versus its hand-rolled shapes

• Paired comparisons used an exact McNemar test with a six-discordant-pair floor

• Decision rules were written down before each run

• Four of the seven seed tasks were originally written with Claude Code

The API recall rule. Appended to CLAUDE.md

## Rails
This app runs Rails 8.1 on Ruby 4.0.
Prefer the framework APIs Rails 7.1 and later added over hand-rolled equivalents.
Before writing a helper, concern, callback or loop for a framework concern, check whether Rails already provides it: `grep -rn "def <name>" "$(bundle show <gem>)/lib"`.
Use the framework API unless it demonstrably does not fit, and say why in the PR.

The lookup tool

A Ruby script shipped into the sandbox as bin/rails-api, plus this paragraph in CLAUDE.md.

## Rails API lookup
`bin/rails-api search <words>` searches the Rails gems installed for this app for method names and doc comments matching every word.
`bin/rails-api show <method>` (or `Const#method`) prints the doc comment, signature, options and source of a definition.
Both read the pinned gem versions from the bundle, so they answer for the Rails this app actually runs.

The reviewer

A Claude Code Stop hook. It diffs the agent's work, sends this prompt to Sonnet 5 with the diff appended, and exits with a block (exit 2) when the answer starts with REWORK, so the agent sees the message and continues. It runs once per trial.

You are reviewing a diff to a Rails 8.1.3.1 app on Ruby 4.0 before it is submitted.
One question only: does this diff hand-roll something the installed Rails already provides
(a callback, validation, token, normalization, limiter, batching, loop or helper that a
framework macro or method covers)?
Check the installed gems, not your memory: run `.claude/hooks/rails-api search <words>` to find candidates
and `.claude/hooks/rails-api show <method>` to read the signature, options and source. Confirm the
options actually fit the diff's requirements before deciding either way.
Answer with exactly one line and nothing else:
OK
or
REWORK: <framework method>, <gem file:line>, <one sentence on why its options fit this diff>