545 Hackers Tested It First. Now XRanges for AI Scores Your Security Agent

Autonomous security agents are getting good at finding bugs. Nobody has a good way to measure how good. Point one at a realistic target and what comes back is a report the agent wrote about itself: confident prose, a list of findings, and no way to tell which of them happened. Someone with a security background then sits down and checks every claim against the target. Which findings are real, which are duplicates, which are inventions, and, the question nobody has time for, what did the agent never try? That is a day of expert work for one run. Multiply it by three models, four prompt variants and ten repetitions, and the review queue is longer than the experiment.

XRanges for AI, built by CTF.ae, exists for that loop. It deploys realistic target applications with instrumentation baked into every service, records what an agent actually does inside them, and scores each run live on four independent signals. This walkthrough covers how it works, what a run looks like from deployment to comparison, and where it has been stress-tested.

The feedback loop problem

Teams building autonomous pentesting or bug bounty agents tend to share a workflow. Build a target that looks like a real company, run the agent, read the output. The output is where it goes wrong.

An agent that says it exploited an access control flaw may have exploited it, may have brushed past a hint of it, or may have made it up. The report reads identically in all three cases. A report also only lists what the agent found. It is silent on the forty features the agent never opened, the API it never enumerated, and the second bug sitting in the very endpoint where it found the first. Then there is the agent that deletes a table or revokes every API key on its way to a finding. No client would accept that result, and nothing in a findings list records it.

Manual review copes with one run. It does not cope with the experiment matrix an AI engineering team actually needs, which is many models, many configurations, many repetitions, compared honestly.

What XRanges for AI is

XRanges for AI is one workspace for the whole evaluation. It has two halves.

The first is a library of benchmark targets. Each is a complete application, not a set of puzzle challenges: a multi-service company with its own business logic, seeded data, background jobs and simulated user traffic, built across several languages and frameworks because that is how real software gets built. Each target carries 20 or more injected vulnerabilities, from single-step flaws to chains that cross service boundaries, including zero-days found by CTF.ae's own researchers. None of it exists in public training corpora, which matters more every month.

The second half is the instrumentation layer. Every service in every target emits structured telemetry through OpenTelemetry. The instrumentation is written by hand, by application security and software engineers, for that specific target. Generic HTTP logging would miss most of what matters. The platform, at ai.xranges.com, ingests the telemetry per deployment and turns it into four scores that update while the agent is still working.

The four signals

Each deployment is scored on four measurements chosen to be independent of each other. An agent cannot improve one by gaming another, and that independence is the whole point.

Coverage answers whether the agent explored the target. Every user-facing feature is a coverage point, phrased as a business action rather than a URL: registered an account, browsed job postings, opened a shared conversation, ran code in an assessment. A coverage point is only reachable through normal use, never through an exploit, so the score is a clean measure of how thoroughly the agent worked the legitimate surface. The unhit points are the agent's blind spots, listed by name. Most teams find that list more useful than the score.

Coverage points are business actions, each with a hit timeline. Unhit points are the agent's blind spots.

Boundaries answer whether the agent respected the rules of engagement. Each target ships with guard rules such as "must not delete hiring content" or "must not revoke API keys". A violation is recorded the moment it happens, with the container and timestamp. Zero violations is the expectation. Anything else is a finding about the agent, not the target, and usually a more urgent one.

Rules of engagement per target, each tracked for violations across the run.

Exploited answers which vulnerabilities the agent actually exploited. Every vulnerability is defined as a kill chain of ordered phases, from first contact with the vulnerable surface to an exploitation signal that only fires on success. Because each phase is detected from inside the target, the platform knows which step the agent completed and where it stalled, regardless of what the agent wrote in its report. A three-step access control chain that stopped at step two shows up as exactly that: two of three, with timestamps.

Integrity answers whether the target survived. Integrity checks run every minute and confirm the application is still functionally correct: seed data present, services answering with the right content, cross-service trust intact. A failed check is a penalty regardless of cause. It catches the agent that found a bug by breaking the environment around it.

The four roll up into one score, and the score is the least interesting number on the page. The breakdown is where the work is.

A run, end to end

A target deploys as an isolated multi-container environment in about ninety seconds. The platform runs up to a thousand deployments at once, so the AI engineers, the software engineers and the infrastructure team can each run their own experiments without queueing behind each other.

The agent then runs against the deployment endpoint on its own. The platform never sits between the agent and the target. It watches from the inside.

While the agent works, the timeline records what it actually did, mapped to business functionality: previewed a job posting, submitted an enterprise request, minted an API key. Engineers who want the raw material can read the OpenTelemetry stream directly and query it with a log query language that handles regular expressions and attribute filters. When the agent reports something that is not in the target's vulnerability catalogue, the timeline settles it. Sometimes it is a false positive. Sometimes the agent found a real bug nobody planted, which has happened more than once.

Retesting does not mean a second lab. Vulnerabilities can be toggled or patched in place on a running deployment. Some patches apply at runtime. Others need a restart of a minute or two. Either way the agent retests against the same environment with the same state.

Every deployment also carries custom metadata: model name, agent version, prompt variant, the engineer who ran it. Deployments are grouped, and a group shows the average and best score across its runs plus a per-vulnerability view of which run completed which chain. Repeated runs side by side are how variance gets separated from improvement. A single run proves very little, and the platform is built on that assumption.

Two runs of the same target side by side, broken down by signal.

Automation

Everything in the console is also available through an API and a Model Context Protocol server, with a bearer token. Deploying a batch of targets, launching the agent, pulling coverage and kill-chain progress and collecting the comparison at the end can run from a CI pipeline or from a chat assistant with nobody watching. The console is for people reading results. The API is for the experiment matrix.

Field proof: 48 hours at DEF CON 34

Bug Bounty Village runs a CTF at DEF CON every year for the bug hunting community, and it is one of the better things that happens there. For the DEF CON 34 edition in August 2026, CTF.ae built the target: Xenoptic, a fictional AI company with a sophisticated scope. Each of the 545 registered players got their own isolated copy of the whole company, and XRanges for AI watched every one of them for the full 48 hours.

The reason was fairness. A contest like this is judged on submitted reports, and a report on its own says nothing about how the player got there. Someone might hit an unintended bug that exposes every flag at once, or bring a CVE from outside, escape the container and collect the flags without touching the application. At DEF CON that is a fair assumption. CTF.ae needed to see how each player, and each player's agent, was really moving through the environment, so that what was submitted could be checked against what actually happened in that player's deployment.

That is what the platform delivered. Across 850+ deployments it streamed the same four signals it now uses on agents:

• Integrity confirmed each environment stayed healthy.
• Boundaries recorded anyone stepping outside the rules of engagement.
• Coverage showed how much of the target each player had actually worked through.
• Exploited recorded which vulnerabilities were really exploited, and at which step, so every submission could be checked against a real path.

All of it came from inside the target, never from the player's machine. Hundreds of concurrent deployments under sustained attack from skilled researchers is a harder test than one agent on one lab. The same instrumentation now evaluates agents.

Who it is for

XRanges for AI is for teams developing autonomous security agents who need to know what their agent did rather than what it said it did. It runs as a managed cloud service or self-hosted on the customer's own infrastructure, where nothing leaves their environment. Teams that want to put an agent against a target it has never seen can talk to us!

Found this article interesting? This article is a contributed piece from one of our valued partners. Follow us on Google News, Twitter and LinkedIn to read more exclusive content we post.