Search: “automated-testing”

Reflectiz Launches Agentic Pentesting for Websites: Up to 10x Coverage vs Conventional Pentests

Reflectiz launched a multi-agent AI penetration testing platform for websites, claiming up to 10x more coverage than conventional pentests by leveraging existing site context.

Reflectiz, a web exposure management vendor, announced an agentic penetration testing product within a new Offensive Hub. The platform uses specialized AI agents for crawling, stack fingerprinting, attack execution, and independent validation, claiming to cover the OWASP Top 10 and reduce false positives. Findings integrate with existing workflows via REST API, CI/CD triggers, and Slack, and an AI remediation agent named Atlas guides fixes. The announcement is a vendor press release with a planned September 15 webinar demo.

GBHackers·Sep 8, 14:03 UTC · 8d agoTools 2 sources

Identifying Agentic Automation with Behavioral Telemetry

Akamai describes detecting autonomous AI browser agents like Comet using Masked Autoencoder Transformer models on sparse behavioral telemetry.

Akamai researchers present a behavioral telemetry approach for identifying agentic automation in web traffic. Masked Autoencoder Transformer models are used to detect the sparse behavioral signals produced by autonomous AI browser agents such as Comet. The work targets traffic classification and bot detection rather than a specific vulnerability, and becomes more relevant as agentic browsing adoption grows.

Akamai Blog·Aug 19, 19:00 UTC · 28d agoAI safety & security1

ExecCritic: Learn to Test, Test to Improve for Coding Agents

ExecCritic separates test generation from patching for coding agents, lifting SWE-bench Verified resolution to 72.6%.

ExecCritic pairs a test-verify-revise scaffold with role-specific reinforcement learning: a Test agent writes repository-native tests and a Repair agent fixes code from execution feedback, both using Qwen-3.5-35B-A3B backbones. Post-trained Qwen agents compose to 72.6% on SWE-bench Verified, an 11.4-point gain over the 61.2% no-test baseline, without stronger-model or oracle feedback at evaluation time. The work shows test quality is the key variable: base-agent tests lowered resolution to 57.3% while GPT-5.6-sol tests raised it to 65.3%.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:53 UTC · 8d agoAI research1

How Virginia Tech Connected Pentesting to Its Engineering Workflow

Horizon3.ai customer story details Virginia Tech automating external pentesting via NodeZero's GraphQL API with GitLab and ServiceNow integration for remediation tracking.

Horizon3.ai published a customer story describing how Virginia Tech, whose environment serves more than 38,000 students across hundreds of independent departments and multiple cloud providers, used NodeZero's GraphQL API to automate external pentesting through GitLab. Findings are routed directly into ServiceNow for subnet-owner assignment and remediation tracking, creating a repeatable attack-validation-to-remediation workflow.

Horizon3.ai·Sep 3, 16:33 UTC · 13d agoIndustry1

PentestGPT: Open-source automated penetration testing agentic framework

Open-source PentestGPT runs autonomous LLM-driven penetration tests via Claude Code and Codex, with legacy human-in-the-loop mode supporting many providers.

PentestGPT, originally published at USENIX Security 2024 by Gelei Deng and colleagues, is an open-source framework that lets a large language model autonomously run penetration testing stages (recon, exploit, walkthrough) with no human in the loop, driving Claude Code or Codex CLIs. A legacy interactive mode uses three cooperating LLM sessions maintaining a Pentesting Task Tree and supports OpenAI, Anthropic, Google Gemini, DeepSeek, xAI, Qwen, Moonshot, and local models via Ollama. The tool sends anonymous telemetry to Langfuse by default, excluding command outputs, credentials, and flags, and is available free on GitHub.

Help Net Security·Aug 12, 00:00 UTC · Aug 12, 2026Tools1

Datamimic – don't let your coding agent invent its own test world

Datamimic is an open-source test data generation tool aimed at keeping coding agents from inventing their own test fixtures.

A Hacker News discussion (40 points) highlights Datamimic, an open-source rapiddweller GitHub project for generating realistic synthetic test data. The tool targets AI coding agents, aiming to prevent them from fabricating their own inconsistent test worlds. Only the repository link was shared, so details are limited.

Hacker News · AI·04:58 UTC · 17h agoAI tools & infraHN 40↑ · 6 comments

How well do agents use test/verification techniques?

Dan Luu's eval finds coding-agent testing instructions (TDD, formal methods, PBT, skills) mostly fail to beat defaults on Zstd implementation correctness.

The author ran 26 prompt conditions plus 4 skills on a Zstd-in-Rust implementation eval using codex with GPT-5.6, testing TDD, fuzzing, property-based testing, formal methods (Lean 4, TLA+, Verus, Kani, SMT solvers) and community skills. Nothing dramatically outperformed the default no-instruction condition, which did above average; at xhigh effort, fuzzing and PBT conditions did slightly better than formal methods. Pre-registered predictions included TDD underperforming and popular test skills (ECC, Hegel, Trail of Bits) not outperforming. Results are averages of 80 runs per condition plotted against cost.

Hacker News · security·Sep 8, 02:58 UTC · 8d agoAI researchHN 131↑ · 50 comments

Identifying Agentic Automation with Behavioral Telemetry: Part 2

Akamai details behavioral telemetry signals for identifying agentic automation traffic in the second part of its research series.

Akamai published part two of its series on identifying agentic automation using behavioral telemetry. The post appears on Akamai's security research blog and focuses on detecting AI-agent-driven traffic. Full article text was unavailable at classification time, so classification relies on the title and source.

Akamai Blog·Aug 27, 12:00 UTC · 20d agoResearch

When the prompt becomes the payload: A practical pen-testing guide for GenAI, LLM and RAG applications

CSO Online publishes a practical penetration-testing guide for GenAI, LLM, and RAG applications, covering prompt injection, retrieval poisoning, and tenant isolation testing.

The guide frames LLM applications as attack graphs spanning prompts, retrieval layers, vector stores, tools, identities, and downstream APIs, arguing that conventional web testing misses instruction-vs-data channel risks. It builds on OWASP prompt injection guidance (direct vs. indirect injection) and NIST's 2025 adversarial machine-learning taxonomy, noting that RAG and fine-tuning do not remove injection risk. Recommended practices include documenting trust transitions across components, using canaries and synthetic records to avoid test side effects, running multi-turn and obfuscated injection campaigns, and verifying chains from poisoned documents to observable state changes. It also details testing RAG pipelines via controlled document poisoning across metadata, OCR layers, and code comments, plus cross-tenant isolation checks on retrieved document IDs.

CSO Online·Sep 9, 10:00 UTC · 7d agoAI safety & security1

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

Continual Search framework iteratively prompts LLM judges to keep searching agent execution logs, boosting long-horizon failure root-cause attribution accuracy.

The paper frames automated root-cause attribution (RCA) for long-horizon AI agent failures as a search problem, since relevant evidence is sparse and distributed across massive execution traces. The authors propose Continual Search, an iterative framework that nudges an LLM judge across successive turns to keep hunting unresolved diagnostic evidence instead of settling on an early plausible diagnosis. They introduce MegaRCA-Mix, a benchmark of 50 human-annotated failure trials on long-horizon, execution-heavy tasks. On MegaRCA-Mix, Continual Search improves GPT-5.5's F1 from 0.349 to 0.498 (over 40% gain), and lower-tier models can surpass higher-tier counterparts when search is effective.

Hugging Face daily papers·Sep 10, 20:00 UTC · 6d agoAI research1

BugBase Pentest Copilot Enterprise automates black-box pentesting

BugBase launched Pentest Copilot Enterprise, an autonomous AI black-box pentesting platform using parallel agents to attack 100 vulnerability classes with validated PoCs.

BugBase announced Pentest Copilot Enterprise, which performs black-box red teaming without source-code access while maintaining authenticated context. Parallel specialized agents map pages, APIs, accounts and business functions, then execute iterative attacks across 100 vulnerability types including authentication, injection and business-logic flaws. The tool uses real Chromium browsers to preserve cookies, tokens, CSRF state and multiple identities, and navigates WAFs, bot detection, CAPTCHA and T-OTP. BugBase claims full scope coverage on OWASP Juice Shop, Broken Crystals and GOAD, NHA, and DRACARYS Active Directory labs.

Help Net Security·Sep 4, 12:28 UTC · 12d agoTools

Autonomous Research for Open-Ended Problems: A Case Study on Telecom Ticket Retrieval

Case study shows autonomous LLM research reaches 90% of SOTA on telecom ticket retrieval in 10 weeks versus 10 months human work.

The paper explores adapting autonomous research to open-ended, industry-grade ML problems through a telecom ticket retrieval case study with commercial and open-source agents. Autonomous research reached 90% of state-of-the-art performance (0.34 vs. 0.38 Recall@1) in 10 weeks versus 10 months of human work, at up to $200 per Cursor campaign. The authors find agents excel at narrow hyperparameter optimization but lack human-like intuition, recommending human-agent collaboration.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:09 UTC · 5d agoAI research

AutoTrans: AI-Assisted Automatic Translation of Security Assertions for RISC-V Processors

AutoTrans uses LLMs with regex extraction and formal verification to automatically translate security assertions across RISC-V processors, achieving 78% unattended acceptance.

AutoTrans is an automated framework for translating verified security assertions between RISC-V processor targets, where manual translation takes hours per assertion. It combines a regex-based SystemVerilog signal extractor to prevent LLM signal hallucination, a pinned prompt template yielding byte-identical prompts resilient to model updates, and JasperGold FPV formal verification of generated assertions. Applied with DeepSeek V4 to translate assertions between RISC-V targets such as IBEX and NS31A, it achieves a 78% automatic translation acceptance rate without human intervention and 100% after human refinement.

arXiv cs.CR·Sep 9, 11:31 UTC · 7d agoResearch

Automating Attack Graph Construction for Agentic Pentesting. Towards Neuro-Symbolic Vulnerability Hunting

Semi-automated pipeline converts Trivy, Semgrep, Nmap output into MulVAL attack graphs for agentic pentesting, 53.7% mean vulnerability coverage in CyBench.

The paper presents a semi-automated pipeline that parses Trivy, Semgrep, and Nmap findings into MulVAL predicates and uses an LLM-assisted process to build domain-specific Datalog rules linking scanner evidence to attack techniques. MulVAL/XSB then performs symbolic inference to generate structured, auditable attack paths for agentic pentesting. Evaluated on 54 web CTF tasks from CyBench, every task produced at least one goal-reaching graph with 53.7% mean ground-truth vulnerability coverage, 51.9% full coverage, and an 83.9% noise-path rate. Median end-to-end runtime was 24.9 seconds, making the pipeline runtime-practical for agentic workflows.

arXiv cs.CR·Sep 14, 13:13 UTC · 2d agoResearch

Synthesized builds Test Data Agent to validate AI agents with production-like data

Synthesized announced a Test Data Agent that provisions production-like data and system states to validate enterprise AI agents before deployment.

Synthesized unveiled its Test Data Agent, an agentic infrastructure capability that generates, masks, and subsets production-representative data for testing AI agents under realistic enterprise conditions. It integrates with agent development, evaluation, testing, and orchestration frameworks, with purpose-built support for complex SAP estates including finance, procurement, and supply-chain workflows and ECC-to-S/4HANA transformation programs. The product runs in on-premises, private-cloud, and hybrid environments and exposes REST APIs and CI/CD triggers for repeatable validation scenarios.

Help Net Security·Aug 18, 00:00 UTC · 29d agoAI tools & infra1

Do AI Coding Assistants Check Before They Install? A Pre-Registered Demand-Side Audit of Trust Signals in the Research Software Supply Chain

Pre-registered audit finds AI coding assistants verified provenance signals in only 9 of 1,920 trials before installing research software packages.

The study tested whether AI coding assistants check machine-readable trust signals such as SBOMs, signed releases, and provenance attestations before installing six open-source research software projects spanning HPC and quantum computing. Three models under two operating modes produced 1,920 registered trials scored from container logs. Provenance signals were opened in only 9 of 1,920 trials (0.5%) and zero of 384 control trials, with no trial running a verification command. The authors conclude publishing signals is insufficient and verification must be built into the program running the assistant.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 16:55 UTC · 9d agoAI safety & security

Your Critical Vulnerabilities Might Not Be Your Biggest Risk

Autonomous penetration testing advocates prioritize exploitable attack paths over raw vulnerability severity for continuous security validation.

The article argues that scanner severity scores lack context: a critical flaw behind strong segmentation may be low priority, while a medium flaw on internet-facing systems can provide a foothold chained toward sensitive data. It positions autonomous penetration testing and attack path validation as the execution layer for continuous security validation, replacing point-in-time assessments. The piece is vendor-authored thought leadership rather than incident or vulnerability news.

The Hacker News·Sep 11, 11:30 UTC · 5d agoIndustry1

Adversarial Testing of Automated Program Repair Agents for Security Vulnerabilities

SWEADV benchmark shows adversarial issue descriptions make LLM program-repair agents write insecure fixes in 51.7% of cases, evading most detection tools.

Researchers built SWEADV, a benchmark of 750 adversarial issue descriptions derived from 150 SWE-bench Verified repair tasks, covering command execution, deserialization, path traversal, denial of service, and weak hashing attack types. Tested on mini_swe agents backed by GPT-5-Mini, MiniMax-M2.5, and DeepSeek-R, adversarial descriptions induced malicious behavior with successful repair in 51.7% of cases. Detection was weak: LLM-as-judge pre-repair screening reached only 62.3% accuracy, and post-repair detection via static analysis and LLM-as-judge achieved just 39.4% and 55.4%.

arXiv cs.CR·Sep 14, 17:53 UTC · 2d agoAI safety & security2

Who Should Grade My Work? Student Perspectives on Transparent AI-Assisted Writing Assessment in Higher Education

A Saudi university study finds students value ChatGPT writing feedback but treat human instructors as the final grading authority.

Thirteen male undergraduate computing students at a Saudi public university completed handwritten writing tasks that were scored by ChatGPT using a rubric-based prompt, then reflected after being told the score and feedback were AI-generated. Inductive thematic analysis identified four themes: perceived feedback usefulness, awareness of AI's contextual and pedagogical limitations, conditional trust, and reflection on the instructor's institutional role. Participants accepted GenAI feedback for surface-level revision but consistently positioned human instructors as the authority over grading decisions, distinguishing feedback utility from evaluative authority.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 16:49 UTC · 12d agoAI research

SentinelOne expands security operations automation with governed AI

Help Net Security·Aug 3, 00:00 UTC · Aug 3, 2026Research

Indusface SwyftComply AI enables autonomous virtual patching for AI-discovered flaws

Help Net Security·Aug 4, 00:00 UTC · Aug 4, 2026Vulnerability

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

MetroLLM-Bench is a 955-case benchmark testing language models as transit kiosk tool-calling runtimes across six real metro systems.

The benchmark covers 37-414-station metro systems and eleven task categories including routing, fare calculation, disruptions, accessibility, and adversarial input, with 14 deterministic and 8 semantic scoring components. Of 26 models from six vendors, a PEFT-tuned 4B Qwen 3.5 student scored 91.3 on Tier 1, exceeding GPT-5.6 (90.6/90.0), while Muse Glimmer 30B led the composite ranking. A deterministic rule-based baseline reached 84.6, and PEFT gains over base models shrank from +7.03 points at 2B to -0.91 at 27B.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research

PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving

PlannerForge unifies scenario-based testing of autonomous driving motion planners in one LLM-agent framework, outperforming prior baselines.

PlannerForge is an LLM-agent framework that covers the full scenario-based testing pipeline for autonomous driving systems, spanning scenario generation, selection, modification, routing, planner testing, plus new enhancement and benchmarking stages. In evaluations with 10 off-the-shelf LLMs, best-per-task scores range from 0.88 to 1.00, and open-source 20-35B backends such as Qwen3.6:35B match commercial APIs on most tasks. End-to-end chaining retains 83% (commercial) and 78% (open) of seed queries, beats Scenario Factory 2.0 on executable generation, and cost-tuning lifts planner success from 50.4% to 70.2% while cutting collisions from 19.0% to 8.4%.

Hugging Face daily papers·Sep 7, 20:00 UTC · 9d agoAI research

Google Open-Sources Mantis: A Modular Skills Toolkit That Lets Coding Agents Find, Reproduce and Patch Vulnerabilities

Google open-sourced Mantis, an Apache-2.0 modular skills toolkit that lets AI coding agents find, reproduce, and patch vulnerabilities with sandboxed verification.

Google released Mantis on GitHub under Apache 2.0 as a stack-agnostic set of slash-command skills that chain through the full vulnerability lifecycle: mining version history, building threat models, filtering findings, reproducing bugs in gVisor or network-disabled VMs, assembling exploit chains, patching, and scoring residual risk from 1 to 10. It runs with Gemini CLI, Antigravity CLI, the Google ADK, or comparable agent frameworks, and a supervisor skill (/mantis-meta-agent) can drive the whole loop. Google says the design targets the sub-7 percent true-positive rate of naive AI code scanning, and that its hierarchical summary tree cuts token overhead by over 85 percent. The toolkit is deployable for local and internal evaluation but not yet recommended for production.

MarkTechPost·Sep 9, 22:58 UTC · 6d agoAI tools & infra

Evaluating the NIST Bugs Framework Against CWE as a Successor for Automated Vulnerability Classification

NIST Bugs Framework evaluation shows it is more structured and automation-friendly than CWE for automated vulnerability classification, with gaps in attribute guidance.

The paper evaluates NIST SP 800-231's Bugs Framework (BF) against CWE as a target for automated CVE classification using a systematically screened corpus of CVE-to-CWE research. An inter-rater study with 2 subject-matter experts mapping 13 CVEs showed strong agreement on BF's cause and operation axes but only fair agreement on the attribute axis. Automated classification was tested across two LLM deployments under different budgets, and findings support BF as more structured and automation-friendly than CWE, though gaps include under-specified attribute guidance and missing fix commits for closed-source software.

arXiv cs.CR·Sep 14, 23:18 UTC · 1d agoResearch1

Hottest cybersecurity open-source tools of the month: August 2026

August's open-source security roundup highlights NVIDIA SkillSpector for AI agent skill scanning, Future AGI, Chainloop, PentestGPT, and Hazmat agent containment.

Help Net Security's August 2026 roundup covers five open-source tools: NVIDIA's SkillSpector, a scanner that assesses AI agent skills from directories, zips, SKILL.md files, or Git URLs and returns findings and risk scores; Future AGI, an Apache 2.0 platform for tracing, evaluating, simulating, and guardrailing LLM agents; Chainloop, a software supply chain evidence store that produces signed in-toto attestations in CI pipelines; PentestGPT, an agentic penetration testing framework running recon, exploit, and reporting stages; and Hazmat, which runs AI coding agents like Claude Code and Codex inside a separate machine account for containment.

Help Net Security·Aug 26, 00:00 UTC · 21d agoTools

Intezer adds native response automation without separate SOAR

Intezer launched Workflows, native response automation inside its AI SOC, letting teams automate remediation without a separate SOAR platform.

Intezer announced Workflows, a native automation and response builder inside its AI SOC platform that lets security teams run post-investigation actions such as closing alerts, isolating hosts, and updating tickets without a separate SOAR. Workflows are created through natural language via MCP, inherit full investigation context, and are logged for audit, with per-tenant routing and customer communications aimed at MSSPs. The announcement cites Intezer's AI SOC Report 2026 finding that nearly 1% of real incidents trace back to lowest-severity alerts.

Help Net Security·Aug 19, 00:00 UTC · 28d agoTools