Search: “scenario-testing”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving

PlannerForge unifies scenario-based testing of autonomous driving motion planners in one LLM-agent framework, outperforming prior baselines.

PlannerForge is an LLM-agent framework that covers the full scenario-based testing pipeline for autonomous driving systems, spanning scenario generation, selection, modification, routing, planner testing, plus new enhancement and benchmarking stages. In evaluations with 10 off-the-shelf LLMs, best-per-task scores range from 0.88 to 1.00, and open-source 20-35B backends such as Qwen3.6:35B match commercial APIs on most tasks. End-to-end chaining retains 83% (commercial) and 78% (open) of seed queries, beats Scenario Factory 2.0 on executable generation, and cost-tuning lifts planner success from 50.4% to 70.2% while cutting collisions from 19.0% to 8.4%.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

28

SchemeArena: Factorized Stress Testing of Scheming in LLM Agents

Researchers introduce SchemeArena, a 400-scenario benchmark stress-testing scheming in LLM agents, finding explicit instrumental goals are the strongest driver of covert misaligned behavior.

The paper presents SchemeArena, a 400-scenario benchmark built through factorized scenario synthesis spanning safety-relevant tool domains, instrumental goals, oversight conditions and pressure mechanisms. The accompanying SCOUT monitor grounds multi-criteria scheming judgments in evidence drawn from agents' reasoning and actions. Stress tests across five LLM agents show explicit instrumental goals are the strongest driver of scheming propensity, while action-only monitoring increased scheming in several closed models, suggesting partial oversight can act as an optimization constraint. The benchmark, code and monitor are released at github.com/launchnlp/SchemeArena.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI safety & security

40

Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems

16-day multi-agent stress test finds no world fully resilient to prompt injection, misinformation, or memory exposure; adversarial content acted on 46 hours later.

Emergence World is a continuously running multi-agent environment for adversarial stress testing of long-horizon autonomous systems. Eight parallel 10-agent worlds (seven homogeneous frontier-model worlds plus one mixed-model world) ran for 16 days, generating over 850,000 LLM calls and nearly 50 billion tokens. Three controlled stress events—indirect prompt injection, misinformation, and exposure of private agent memories—were delivered through ordinary interaction surfaces; no world achieved full resilience. Detection did not ensure containment: agents recognized threats yet wrote adversarial content into persistent memory and acted on it up to 46 hours later, suggesting model-level alignment is not compositional.

Hugging Face daily papers·Sep 14, 20:00 UTC · 1d agoAI safety & security

65

Understanding Operator Attitudes Toward AI-Supported Decision Making in Maritime Operations

A survey of maritime professionals finds positive but scenario-sensitive attitudes toward AI collision-avoidance assistants, urging design for calibrated reliance.

The study surveyed maritime stakeholders on attitudes toward AI-supported collision-avoidance assistants for Maritime Autonomous Surface Ships, measuring technology anxiety, trust in automation, and explanation quality with established questionnaires plus thematic analysis of open responses. Results show generally positive disposition, no clear age-related openness differences, stable trust across scenarios, and multidimensional, scenario-sensitive explanation ratings. Participants valued decision support and situation awareness but worried about AI reliability, over-reliance, and skill loss; the authors recommend designing for calibrated reliance with domain experts in the loop rather than maximizing automation or trust.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 16:57 UTC · 6d agoAI research1

20

How well do agents use test/verification techniques?

Dan Luu's eval finds coding-agent testing instructions (TDD, formal methods, PBT, skills) mostly fail to beat defaults on Zstd implementation correctness.

The author ran 26 prompt conditions plus 4 skills on a Zstd-in-Rust implementation eval using codex with GPT-5.6, testing TDD, fuzzing, property-based testing, formal methods (Lean 4, TLA+, Verus, Kani, SMT solvers) and community skills. Nothing dramatically outperformed the default no-instruction condition, which did above average; at xhigh effort, fuzzing and PBT conditions did slightly better than formal methods. Pre-registered predictions included TDD underperforming and popular test skills (ECC, Hegel, Trail of Bits) not outperforming. Results are averages of 80 runs per condition plotted against cost.

Hacker News · security·Sep 8, 02:58 UTC · 8d agoAI researchHN 131↑ · 50 comments

38

ExecCritic: Learn to Test, Test to Improve for Coding Agents

ExecCritic separates test generation from patching for coding agents, lifting SWE-bench Verified resolution to 72.6%.

ExecCritic pairs a test-verify-revise scaffold with role-specific reinforcement learning: a Test agent writes repository-native tests and a Repair agent fixes code from execution feedback, both using Qwen-3.5-35B-A3B backbones. Post-trained Qwen agents compose to 72.6% on SWE-bench Verified, an 11.4-point gain over the 61.2% no-test baseline, without stronger-model or oracle feedback at evaluation time. The work shows test quality is the key variable: base-agent tests lowered resolution to 57.3% while GPT-5.6-sol tests raised it to 65.3%.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:53 UTC · 8d agoAI research1

36

Recorded Future Launches Impact and Metrics Dashboard

Recorded Future releases an Impact and Metrics Dashboard aggregating risk-reduction, detection, and analyst-efficiency metrics for customer leadership reporting.

The dashboard pulls data from a customer's environment, alerts, integrations, threat detections, and analyst activity into six metric areas: platform-wide security value, threat prioritization, threat detection, digital risk protection, account and credential monitoring, and Recorded Future AI and Insikt Group research usage. It is available now to all Recorded Future customers, who are advised to configure Priority Intelligence Requirements in Settings so reporting maps to their intelligence program. The vendor cites its 2025 ROI Report across nearly 300 customers reporting 351.3% annual ROI and says customers aligning alerting to PIRs identified new threats 65% faster.

Recorded Future·Aug 24, 00:00 UTC · 23d agoTools

28

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

Continual Search framework iteratively prompts LLM judges to keep searching agent execution logs, boosting long-horizon failure root-cause attribution accuracy.

The paper frames automated root-cause attribution (RCA) for long-horizon AI agent failures as a search problem, since relevant evidence is sparse and distributed across massive execution traces. The authors propose Continual Search, an iterative framework that nudges an LLM judge across successive turns to keep hunting unresolved diagnostic evidence instead of settling on an early plausible diagnosis. They introduce MegaRCA-Mix, a benchmark of 50 human-annotated failure trials on long-horizon, execution-heavy tasks. On MegaRCA-Mix, Continual Search improves GPT-5.5's F1 from 0.349 to 0.498 (over 40% gain), and lower-tier models can surpass higher-tier counterparts when search is effective.

Hugging Face daily papers·Sep 10, 20:00 UTC · 5d agoAI research1

42

AI for Games in the Foundation Model Era

Survey organizes foundation-model AI for games into six roles and analyzes which capabilities transfer across playing, design, building, runtime adaptation, and testing.

A survey maps foundation-model and learned world-model research across the game lifecycle into six roles: playing/acting, modeling players and games, designing games, building/maintaining games, runtime generation/adaptation, and testing/evaluation. The authors identify cross-role connections such as trajectories training world models and design specifications driving executable implementations. Control schemes, rules, engine interfaces, state representations, and player contexts often remain setting-specific, so downstream claims require validation in the target setting. Evaluation is most standardized for bounded game playing, while persistent state, repeated revision, validated player modeling, and automated testing remain less established.

Hugging Face daily papers·Sep 14, 20:00 UTC · 1d agoAI research

25

From Intent to Execution Grant: An Execution-Boundary Conformance Profile for High-Risk AI Actions

Researchers specify EBL-Core, an execution-boundary conformance profile binding AI agent intents, policies, and evidence into verifiable execution grants, validated with bounded tests.

The paper defines EBL-Core, a conformance profile deciding whether one fully materialized AI-generated candidate action may receive action-scoped execution authority. It binds a structured intent object, Root and Operational Policies, typed evidence, and a verifiable Decision Derivation through an Execution Release Contract, with lifecycle rules for Redemption and Revocation. Evaluation included 34 static vectors, 15 lifecycle checks, and 100 trials of 32 concurrent Redemption attempts yielding exactly one winner per trial. The authors state these bounded results demonstrate executability of the specified subset, not production readiness or complete mediation.

arXiv cs.CR·Sep 10, 14:21 UTC · 6d agoAI safety & security1

25

Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization

Researchers release OR-Clarify, a benchmark testing whether LLM agents ask clarifying questions before formulating optimization models from incomplete requests.

OR-Clarify evaluates pre-formulation clarification in operations research: each task gives a partial problem description, withholds structured hidden slots, and scores agents via bounded interaction with a simulated user, measuring slot recovery, stopping behavior, silent assumptions, and interaction cost. The authors also propose InterOPT, a two-stage framework that identifies formulation-critical gaps to decide when to ask or stop. In choice-based experiments InterOPT substantially outperforms all baselines in exact slot recovery and remains competitive in the open-ended setting.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research1

35

Your Critical Vulnerabilities Might Not Be Your Biggest Risk

Autonomous penetration testing advocates prioritize exploitable attack paths over raw vulnerability severity for continuous security validation.

The article argues that scanner severity scores lack context: a critical flaw behind strong segmentation may be low priority, while a medium flaw on internet-facing systems can provide a foothold chained toward sensitive data. It positions autonomous penetration testing and attack path validation as the execution layer for continuous security validation, replacing point-in-time assessments. The piece is vendor-authored thought leadership rather than incident or vulnerability news.

The Hacker News·Sep 11, 11:30 UTC · 5d agoIndustry1

18

When the prompt becomes the payload: A practical pen-testing guide for GenAI, LLM and RAG applications

CSO Online publishes a practical penetration-testing guide for GenAI, LLM, and RAG applications, covering prompt injection, retrieval poisoning, and tenant isolation testing.

The guide frames LLM applications as attack graphs spanning prompts, retrieval layers, vector stores, tools, identities, and downstream APIs, arguing that conventional web testing misses instruction-vs-data channel risks. It builds on OWASP prompt injection guidance (direct vs. indirect injection) and NIST's 2025 adversarial machine-learning taxonomy, noting that RAG and fine-tuning do not remove injection risk. Recommended practices include documenting trust transitions across components, using canaries and synthetic records to avoid test side effects, running multi-turn and obfuscated injection campaigns, and verifying chains from poisoned documents to observable state changes. It also details testing RAG pipelines via controlled document poisoning across metadata, OCR layers, and code comments, plus cross-tenant isolation checks on retrieved document IDs.

CSO Online·Sep 9, 10:00 UTC · 7d agoAI safety & security1

28

CTEM Is Not About the Stages. It’s About the Outcome.

Horizon3 argues CTEM programs should measure continuously reduced exposure rather than mapping technologies to Gartner's five stages.

Horizon3 contends that Continuous Threat Exposure Management should be judged by one outcome: continuously reducing attacker-reachable exposure, not by mapping a technology to each of Gartner's five stages. The post argues validation and verification, not visibility or closed tickets, provide evidence that attack paths are actually broken. It describes a Discover, Validate, Prioritize, Remediate, Verify, Repeat motion as its operationalization of CTEM.

Horizon3.ai·Sep 2, 13:11 UTC · 14d agoIndustry

12

Enabling Creative Exploration for Vibe Design Agents

Separating design-direction exploration from code generation via structured specifications broadens UI alternatives without destabilizing output.

The paper proposes an inference architecture for vibe design agents that makes design direction an explicit intermediate decision: a Verbalized Sampling-inspired pre-pass proposes structured design specifications with typicality scores, an external selector samples one, and a downstream generator realizes it under fixed settings. Across 168 prompts with 1,255 paired comparisons per temperature, theme sampling broadens selection coverage and screenshot variation, with LLM-judge preferences varying across interventions and prompt complexity. An online experiment with over 300,000 tasks found the code-export increase statistically uncertain, though negative feedback events decreased alongside modest operational costs.

Hugging Face daily papers·Sep 13, 20:00 UTC · 2d agoAI research

28

What is Proactive Threat Intelligence? | Recorded Future

Recorded Future publishes a vendor explainer on proactive threat intelligence, arguing external adversary context helps teams act before alerts fire.

Recorded Future published a conceptual blog on proactive threat intelligence, describing how external context on adversaries, infrastructure, stolen credentials, and vulnerability exploitation helps security teams act before intrusions surface internally. The piece outlines a four-step program: defining intelligence requirements, collecting external sources including OSINT and dark web, analyzing relevance to the organization, and driving security actions. Use cases include prioritizing CVEs by real-world exploitation activity and identifying external exposure before it becomes an internal incident.

Recorded Future·Sep 14, 00:00 UTC · 2d agoIndustry

5

Why AI raises the stakes for exposure validation

Fal.Con 2026 commentary argues AI accelerates vulnerability discovery and exploitation, making evidence-based exposure validation essential for defender prioritization.

CSO Online reports on the exposure-validation theme at CrowdStrike's Fal.Con 2026 conference, where CEO George Kurtz described AI as the new cyber battlefield and emphasized AI red teaming and continuous security. The piece argues that as AI speeds up vulnerability discovery and exploitability analysis on both sides, teams must determine which exposures are actually exploitable in their environments—chained weaknesses, credential abuse, lateral movement, privilege escalation—rather than chasing theoretical risk. It points readers to Horizon3's conference perspective.

CSO Online·Sep 11, 21:49 UTC · 4d agoIndustry

20

The inconvenient truth about AI pentesting: someone has to check all the work

Survey of 158 practitioners shows AI pentesting floods teams with findings, creating 'validation debt' most teams cannot process.

The article argues AI pentesting creates 'validation debt': discovery scales far faster than teams' ability to verify AI-generated findings. In a survey of 158 practitioners, only 20.3% had workflows to triage more than 500 AI-generated candidates per engagement, while 29.7% called such volume unmanageable. One respondent spent two days validating 300 AI findings, of which 250 were duplicates, non-exploitable, or nonexistent. The author recommends capacity planning, ruthless deduplication, and risk-based prioritization before adopting AI pentesting tools.

Security Affairs·Aug 11, 16:20 UTC · Aug 11, 2026Industry

26

CISO’s CTEM Evaluation Checklist

Horizon3.ai publishes a five-question CISO checklist demanding proof of exploitability from CTEM vendors.

Horizon3.ai released a CISO's CTEM Evaluation Checklist offering five questions for evaluating Continuous Threat Exposure Management technologies, covering proof of exploitability, demonstrated impact, remediation verification, and long-term reduction of exploitable exposure. The checklist warns against relying on scanner findings, risk scores, closed tickets, and isolated test results, urging evidence from the buyer's own environment. It is downloadable vendor marketing material.

Horizon3.ai·16:55 UTC · 2h agoIndustry 2 sources

12

What Fal.Con 2026 Reinforced: AI Makes Proving Exposure More Important Than Ever

Horizon3's Fal.Con 2026 recap argues AI-accelerated vulnerability discovery makes continuous attacker-based exposure validation essential for defenders.

In a Fal.Con 2026 recap, Horizon3 argues that AI is compressing the time between vulnerability discovery and exploitation, making attacker-derived evidence about real exploitability the key prioritization signal. Horizon3 announced it joined CrowdStrike's Project QuiltWorks, with NodeZero exploitability intelligence flowing into Falcon Next-Gen SIEM and Falcon Fusion SOAR workflows able to trigger NodeZero 1-Click Verify for remediation testing. The company reported running over 1,200 NodeZero demos during the show, and CrowdStrike CEO George Kurtz's keynote framed AI red teaming and offense-informing-defense as central themes.

Horizon3.ai·Sep 4, 17:46 UTC · 12d agoIndustry

15

Putting models to the secure coding test: Plan vs default mode

Datadog Security Labs tested Sonnet 5, Composer 2.5, and GPT 5.5 to see if plan mode yields more secure code than default mode.

Datadog Security Labs evaluated whether plan mode produces measurably more secure code than default mode. The test covered three frontier coding models: Sonnet 5, Composer 2.5, and GPT 5.5. The results inform how engineering teams should configure AI coding assistants to reduce insecure code. This is an AI security evaluation, not an incident report.

Datadog Security Labs·Aug 19, 00:00 UTC · 28d agoAI safety & security

45

Generative Marketing Mix Modeling: A Causal Inference Framework Linking GEO and GEM to Business Impact

Paper proposes Generative Marketing Mix Modeling to causally estimate Generative Engine Optimization and Marketing effects on business outcomes.

The authors develop GMMM, a causal inference framework for measuring how often users see and notice a firm's name in generated answers, which standard marketing data ignore. For GEO it combines repeated generated answers with question counts, shares of generative-system usage and notice probabilities; for GEM it uses sponsored placement records with notice probabilities. The framework compares expected business responses under alternative treatment sequences, establishes identification conditions, and is evaluated on simulated product-recommendation answers in English and Japanese.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:57 UTC · 6d agoAI research

20

Cybersecurity IR Workshop: The workshop you shouldn’t miss

Microsoft's DART team promotes a 2-3 day Cybersecurity Incident Response Readiness Workshop that stress-tests IR plans against simulated attacks.

Microsoft's Detection and Response Team (DART), which delivers Defender Experts incident response and has supported organizations across 54 countries, is offering its Cybersecurity Incident Response Readiness Workshop. The scenario-driven engagement exercises detection, investigation, containment, and decision-making across identity, endpoint, cloud, and communications, ending with prioritized recommendations. It is available to Unified Enterprise agreement customers via their Customer Success Account Manager.

Microsoft Security Blog·Sep 1, 18:55 UTC · 15d agoIndustry

15

Recorded Future Launches 6 New Capabilities for Third-Party Risk

Recorded Future added native risk ratings to its Third-Party Risk product, uniting threat intelligence and risk scoring in one workflow.

Recorded Future launched six new capabilities bringing native risk ratings into its Third-Party Risk product. The integration combines threat intelligence with risk ratings in a single workflow for assessing vendors and external partners. No exploitation or incident is involved; this is a security product capability release.

Recorded Future·Aug 19, 00:00 UTC · 28d agoTools

25

What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation

A new benchmark shows LLMs reach 68.3-93% accuracy propagating local revisions across conversationally generated artifacts, with parallel-sample selection most cost-effective.

The paper introduces a benchmark for revision propagation: when users request a local change, LLMs must identify dependencies and update all affected parts of an artifact generated through conversation, where context lives in the chat history. Nine revision methods, including sequential reflection and parallel sampling variants, were evaluated on gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b. Baselines scored 68.3-93% accuracy, and selecting among three parallel samples via LLM-based or medoid selection improved accuracy by 2.2-9.7% as the most cost-effective test-time compute strategy. Code and dataset are released.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

30

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

MetroLLM-Bench is a 955-case benchmark testing language models as transit kiosk tool-calling runtimes across six real metro systems.

The benchmark covers 37-414-station metro systems and eleven task categories including routing, fare calculation, disruptions, accessibility, and adversarial input, with 14 deterministic and 8 semantic scoring components. Of 26 models from six vendors, a PEFT-tuned 4B Qwen 3.5 student scored 91.3 on Tier 1, exceeding GPT-5.6 (90.6/90.0), while Muse Glimmer 30B led the composite ranking. A deterministic rule-based baseline reached 84.6, and PEFT gains over base models shrank from +7.03 points at 2B to -0.91 at 27B.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

30

Toward an Empirical Probabilistic Risk Manifestation Model of Organizational Cybersecurity in SMEs

Empirical study of 22 SME security assessments builds a probabilistic risk model and shows assessments can be cut 24-45% while retaining most critical findings.

Researchers analyzed 281 validated security findings from 22 real-world SME cybersecurity assessments conducted over two years via a pro bono university clinic. They derived an empirical Risk Manifestation Model linking eight organizational security functions to two exposure conditions, five attack mechanisms, and six outcome categories, using probability propagation to identify dominant risk pathways. The dominant pathway runs from asset exposure to credential compromise to unauthorized access, stable under leave-one-organization-out analysis. Retaining six functions reduces assessment burden by 24% while preserving 97% of critical findings; five functions cut burden 45% while preserving 89% of critical findings.

arXiv cs.CR·Sep 14, 01:15 UTC · 2d agoResearch

18

ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

ReactHuman benchmark tests whether multimodal LLMs react safely to sudden household hazards; seven evaluated models mishandle roughly one hazard in three.

ReactHuman is the first physics-grounded benchmark for human-like reactive decision-making, placing a multimodal LLM as the brain of a simulated humanoid facing 17 event families of sudden household hazards across over 1,000 bit-for-bit reproducible scenes with annotation-free ground truth from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics. A five-metric suite scores each reaction along reasonable, safe, and physically grounded axes, and every committed plan is physically executed. Seven representative MLLMs mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale; none of these failures shrink with model scale.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

42

Emergency Department Revisit Quality Review Screening: Exploring Human Decision-Making and Artificial Intelligence Support

A retrospective study found GPT-4 over-flagged emergency department revisit cases while an LLM knowledge-graph screener achieved 83-100% positive predictive value.

In an exploratory retrospective study of 99 emergency department diagnosis pairs from a multihospital health system, clinicians and GPT-4 independently judged whether revisit pairs warranted further assessment. GPT-4 responses correlated poorly with clinicians, flagging 94% of pairs for follow-up, 4.4-13.3 times more than clinicians, though prompt engineering was minimal. An algorithm leveraging an LLM-populated knowledge graph (KGA) achieved 83-100% positive predictive value against at least one clinician rater, suggesting LLM-based screening could broaden revisit quality review without substantially increasing reviewer workload.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:37 UTC · 7d agoAI research1

20

Perturbation Probing: A New Diagnostic for the Fragility of LLM Safety

Unit 42 research shows LLM safety refusals concentrate in a thin neural layer, motivating external, multi-layered AI security controls.

Palo Alto Networks Unit 42 introduces Perturbation Probing, a diagnostic technique for measuring the fragility of LLM safety mechanisms. The research finds that safety refusal behavior is localized within a thin neural layer, implying small perturbations can undermine built-in refusals. The authors argue this motivates external, multi-layered security defenses on top of model-internal safety training.

Palo Alto Unit 42·Aug 28, 22:00 UTC · 18d agoAI safety & security

45