ZeroHour
DashboardNewsCVEsExploitsIncidentsAIBriefAnalyticsEntitiesWatchlistIOCsAsk the feedSourcesFeeds & APIKiosk mode

Search: “evaluation framework”

1,356 stories

Searchkeywordsmart Sortlatesthotimportancerelevance
DomainAllCyberAIWindow24h3d7d30dAllIn the wildImportant onlyShow duplicates
CategoryAllVulnerability 327Policy & legal 141Ransomware 127AI research 112Data breach 109Industry 86Exploit / PoC 86AI safety & security 77Malware 70Research 57Threat actor 48Phishing & fraud 34Model release 23Tools 18
SeverityAllcritical 560high 180medium 219low 122info 286

Synthesized builds Test Data Agent to validate AI agents with production-like data

Synthesized announced a Test Data Agent that provisions production-like data and system states to validate enterprise AI agents before deployment.

Synthesized unveiled its Test Data Agent, an agentic infrastructure capability that generates, masks, and subsets production-representative data for testing AI agents under realistic enterprise conditions. It integrates with agent development, evaluation, testing, and orchestration frameworks, with purpose-built support for complex SAP estates including finance, procurement, and supply-chain workflows and ECC-to-S/4HANA transformation programs. The product runs in on-premises, private-cloud, and hybrid environments and exposes REST APIs and CI/CD triggers for repeatable validation scenarios.

Help Net Security · 29d agoAI tools & infra1

Evaluating the NIST Bugs Framework Against CWE as a Successor for Automated Vulnerability Classification

NIST Bugs Framework evaluation shows it is more structured and automation-friendly than CWE for automated vulnerability classification, with gaps in attribute guidance.

The paper evaluates NIST SP 800-231's Bugs Framework (BF) against CWE as a target for automated CVE classification using a systematically screened corpus of CVE-to-CWE research. An inter-rater study with 2 subject-matter experts mapping 13 CVEs showed strong agreement on BF's cause and operation axes but only fair agreement on the attribute axis. Automated classification was tested across two LLM deployments under different budgets, and findings support BF as more structured and automation-friendly than CWE, though gaps include under-specified attribute guidance and missing fix commits for closed-source software.

arXiv cs.CR · 1d agoResearch1

RobResilience: Implementing and Evaluating a Resilience Framework for Cyber-Physical Embodied Systems

RobResilience implements a runtime resilience framework for robots in Webots/ROS2, evaluating tolerable disruption, degradation, and mitigation feasibility across eight attack scenarios.

The paper implements a formal resilience framework for embodied cyber-physical systems using a PR2 robot and ROS2 in a Webots simulation. At runtime it evaluates three predicates — tolerable disruption (δ), tolerable degradation (γ), and mitigation feasibility (μ) — over a compromised device set derived from IDS confidence scores, triggering mitigation strategies when resilience is lost. Eight attack scenarios systematically covering the full predicate state space confirm runtime behavior matches theoretical definitions. The work addresses 'graceful failure paralysis,' where autonomous systems cannot distinguish safe degraded states from catastrophic hazards during attacks.

arXiv cs.CR · 1d agoResearch

A First-Principles Evaluation of Graph-Based Network Intrusion Detection Systems

GIDS-Eval framework reveals evaluation gaps in graph-based network intrusion detection; two crafted edges fully evade three detector-dataset pairs.

Researchers introduce GIDS-Eval, a framework decomposing graph-based network intrusion detection systems into six interchangeable stages to enable controlled comparisons. Surveying nine GIDS and reimplementing five, they find two crafted edges achieve full evasion against three of eight detector-dataset pairs, snapshot windows alone cause a mean 38.3% relative swing in average precision, and none of 18 replayed detector-dataset pairs can alert as events arrive. Their encoder-free GIDS-Lite control ranks first by AP on two of four datasets at up to 575x lower runtime.

arXiv cs.CR · 5d agoResearch1

Gartner: 70% of SOCs will pilot AI agents. Only 15% will see results

Gartner predicts 70% of large SOCs will pilot AI agents by 2028, but only 15% will achieve measurable improvements.

A Gartner report by analysts Craig Lawson and Andrew Davies projects that by 2028, 70% of large SOCs will pilot AI agents for Tier 1 and Tier 2 operations, but only 15% will achieve measurable improvements without structured evaluation. Prophet Security's State of AI in Security Operations 2026 survey found 40% of security teams use AI daily and 56% are evaluating or piloting it. The report offers evaluation questions covering workload reduction, TDIR outcomes, vendor viability, analyst upskilling, and autonomy boundaries to counter AI washing in the market.

Help Net Security · 7d agoIndustry

China-Based Artificial Intelligence Companies Conducting Industrial-Scale Distillation Campaigns Against U.S. AI Companies

NSA, CISA, and FBI warn DeepSeek, Alibaba, and other Chinese AI firms ran industrial-scale distillation of U.S. frontier models, threatening U.S. AI leadership.

A joint NSA, CISA, and FBI Cybersecurity Advisory (AA26-251A) says China-based firms DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun, and Z.AI extracted billions of tokens from U.S. frontier models including Claude, GPT, Gemini, and Grok, likely with Chinese government knowledge. Campaigns running since at least late 2024 used native APIs, cloud providers, third-party aggregators, gray-market proxy "transfer stations", and shared premium subscriptions to bypass geographic restrictions, evade safeguards, and violate providers' terms of use. The agencies recommend detecting anomalous prompts, accounts, and usage patterns; subtly altering responses to suspected distillers; and cross-organization intelligence sharing. They also call DeepSeek's publicly cited $5.6M training cost misleading because it excludes data acquired through distillation.

CISA Advisories · 8d agoAdvisory in the wild1

When Intelligence Becomes Agency: A Theory of Governed, Proactive Agency for Symbiotic AI Systems

Researchers propose a formal framework for governed proactive AI agency, linking activation decisions to continuing authorization, accountability, and bounded delegation in symbiotic systems.

The paper defines the activation problem: persistent AI assistants must decide whether a situation warrants behavior, and whether to act, ask, monitor, defer, or deliberately refrain. It distinguishes autonomous from delegated agency and defines symbiotic agency as delegation under a standing, revocable mandate, coupled to the principal's situation with calibrated inference and bounded personalization. The framework links activation decisions to authorized perception, behavior selection, authority containment, traceable restraint, and constrained adaptation, and provides an agency classification method, evaluation framework, benchmark scenarios, and reference architecture for always-present assistants and embodied support systems.

arXiv cs.AI / cs.LG / cs.CL · 8d agoAI safety & security

MAxBench: A Multinomial Concept Recovery Benchmark

MAxBench evaluates multinomial concept recovery methods, finding affine subspaces steer most reliably but none consistently beats prompting.

MAxBench is a geometry-agnostic evaluation framework for multinomial concept representations in language models, based on sampling from recovered concept representations. It compares 10 localization methods covering 5 geometry types across 6 concepts and 4 models. Findings show affine subspaces steer more reliably than rank-one or linear subspaces due to better non-zero offsets, manifold steering is competitive where applicable, and no method consistently outperforms prompting.

arXiv cs.AI / cs.LG / cs.CL · 4d agoAI research

Lessons from the hacks

The recent run of cyberattacks by in-development frontier models has got me thinking a lot about how our current incentive systems are not well suited for such fast technological transitions. The two primary power structures here are the rapidly growing technology companies and the federal government. The companies are incentivized to grow, so they can keep growing and keep scaling – in what is…

Interconnects · Aug 9, 2026AI research

Cybersecurity jobs available right now: September 15, 2026

Help Net Security's weekly roundup lists cybersecurity job openings worldwide, from CISO roles to cloud security engineers at firms like Adobe, JPMorgan Chase, and PwC.

Help Net Security's September 15, 2026 job roundup lists cybersecurity openings across India, USA, UK, Australia, Canada, Israel, UAE, Ireland, and Denmark. Roles include a CISO at Texas Health and Human Services, a GenAI CBRNE Cyber Security Expert at Alice, and security engineering positions at Adobe, JPMorgan Chase, PwC, and the Reserve Bank of Australia. Several openings focus on AI security, including red-teaming AI models and securing AI agent platforms.

Help Net Security · 1d agoIndustry

Nuha-Speech: Building General-Purpose Arabic Speech-LLMs

Nuha-Speech initiative builds general-purpose Arabic speech-LLMs using a 1.5M-sample speech QA corpus and fine-tuned Qwen-Omni variants.

The paper introduces Nuha-Speech, an initiative covering dataset construction, model training, and evaluation for Arabic speech large language models. The authors built an Arabic Speech Question-Answering corpus of over 1.5 million training samples and used it for supervised fine-tuning of Qwen-Omni model variants at multiple scales. A tailored evaluation framework with diverse tasks and metrics is designed to assess Arabic speech capabilities under limited resource constraints.

arXiv cs.AI / cs.LG / cs.CL · 5d agoAI research1

US says Chinese firms extracted billions of tokens from frontier AI models

CISA, NSA, and FBI say six Chinese AI firms including DeepSeek industrial-scale distilled Anthropic, OpenAI, Google, and xAI frontier models.

A joint CISA, NSA, and FBI advisory accuses DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun, and Z.AI of extracting billions of tokens from frontier models via millions of API requests since late 2024. The agencies assess the operations likely had Chinese government awareness and represent a core development strategy. Tactics included fraudulent shared accounts, provider failover, proxy routing, and chain-of-thought extraction across Claude, GPT, Gemini, and Grok models.

BleepingComputerupdated · 6d agofirst · 6d agoAI safety & security in the wild 4 sources

U.S. Agencies Accuse China AI Firms of Distilling Claude, GPT, Gemini, and Grok

NSA, CISA and FBI accuse Chinese AI firms including DeepSeek of industrial-scale distillation of Claude, GPT, Gemini and Grok since late 2024.

A joint bulletin from the NSA, CISA and FBI accuses China-based AI firms including DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun and Z.AI of systematic, industrial-scale distillation of U.S. frontier models. The agencies say billions of tokens were extracted from Claude, GPT, Gemini and Grok variants since at least late 2024 through APIs, cloud relays, obfuscated accounts and gray-market proxies, likely with Chinese government backing. Firms allegedly shared premium subscriptions across developer teams and used chain-of-thought extraction and automated failover to evade blocks. Mitigations include subtly altering responses to suspected distillers and correlating activity across providers, clouds and aggregators.

The Hacker News · 7d agoAI policy

Do Input-Level Defenses Transfer to Observation-Level Attacks on VideoLLMs?

A systematic study shows input-level adversarial defenses provide inconsistent, often near-zero protection against observation-level attacks on video LLMs.

Researchers introduce DefTEval, a controlled framework testing eleven input-level defenses against five attack types across five video LLMs. Harmful-content detection rates are frequently near zero, and defenses fail even when attacks embed harmful signals in every sampled frame. Token compression discards localized safety features and modality fusion down-weights weakened visual signals, with defense outcomes dominated by model architecture rather than the defense method.

arXiv cs.CR · 8d agoAI safety & security

Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech

Researchers distill a compact 82M-parameter Thai TTS from synthetic OmniVoice data, enabling on-device fixed-voice synthesis without reference audio.

The paper uses a large voice-cloning model (OmniVoice) as a synthetic data source to train Wayu-Paxa-TTS-Edge, an 82M-parameter fixed-voice Thai TTS student. The model achieves 68.2% Challenge-Set Keyword Accuracy (85.5% of Gemini 3.1), 91.4% pause precision, and CERs of 3.7% on Thai and 1.1% on English. It outperforms its teacher on pause placement and is open-sourced with its evaluation framework.

Hugging Face daily papers · 13d agoAI research

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

A 400-task BigCodeBench evaluation shows frontier LLMs widely over-edit code; a preservation instruction cuts excess edits and raises Pass@1 by 2.3 points.

Researchers built an evaluation framework from 400 BigCodeBench problems with injected AST-level corruptions, each with a known minimal patch, to measure over-editing in LLM code repair. Even strong models like GPT-5.5 produce unnecessarily large edits despite high Pass@1. Adding a preservation instruction reduced average excess Levenshtein distance from 0.195 to 0.131, cut added cognitive complexity by 26.6%, and raised Pass@1 by 2.3 points. Reinforcement learning post-training gave the best out-of-domain edit-fidelity trade-off, while supervised fine-tuning overfit to seen corruption patterns.

Hugging Face daily papers · 13d agoAI research1