ZeroHour
DashboardNewsCVEsExploitsIncidentsAIBriefAnalyticsEntitiesWatchlistIOCsAsk the feedSourcesFeeds & APIKiosk mode

Search: “ai benchmark”

656 stories

Searchkeywordsmart Sortlatesthotimportancerelevance
DomainAllCyberAIWindow24h3d7d30dAllIn the wildImportant onlyShow duplicates
CategoryAllVulnerability 122AI research 72Exploit / PoC 67AI safety & security 67Policy & legal 48Model release 45Ransomware 42Data breach 35Industry 31AI industry 31Phishing & fraud 26Malware 25Research 16Threat actor 12
SeverityAllcritical 235high 49medium 120low 57info 204

Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

Benchmark Radar provides a living searchable database of 1,283 AI benchmark records and 12,916 score observations drawn from 37 daily discovery sources.

Benchmark Radar combines daily discovery of benchmark papers, repositories, datasets, and releases from 13 direct connectors and 24 first-party feeds into a searchable catalog with model card mentions and score histories. The catalog contains 1,283 source records drawn from 4 benchmark catalogs plus 12,916 numeric observations on 790 records. The release includes a web dashboard with leaderboard, Pareto frontier of score versus usage, saturation and trend views, daily feeds, a CLI, and reproducible analysis. The paper audits the full catalog and examines benchmark saturation and limits of score comparisons.

Hugging Face daily papers · 6d agoAI research

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

Real-SWE benchmark tests coding agents on licensed private enterprise codebases; top model Fable 5.1 resolves only 38.8% of tasks.

Real-SWE is a new benchmark evaluating frontier AI coding agents on tasks drawn from private production codebases licensed from real companies, spanning billing, tax calculation, and cross-service migrations. Fable 5.1 with Claude Code leads at 38.8% resolution rate (pass@1 over eight runs), followed by GPT-6 Astra Codex CLI at 33.8% and Gemini 3.8 Flash Gemini CLI at 31.2%. Tasks use native harnesses and realistic tooling including Docker, Kubernetes, PostgreSQL, Redis, and Linear; median reference solutions edit 11 files versus 6 for DeepSWE and FrontierCode.

Hacker News · AI · 3d agoAI researchHN 45↑ · 31 comments

Show HN: Nari Qwen3-TTS and Qwen3-ASR – High accuracy, low latency and cost

Nari Labs claims top Coval voice AI benchmark rankings with low-latency, low-cost Qwen3-ASR and Qwen3-TTS inference endpoints.

Nari Labs says its Qwen3-ASR Fast endpoint ranks #1 in Coval's time-to-final-segment latency (p50 44 ms) with 3.6% WER at $0.12/hour, behind only AssemblyAI Universal 3.5 Pro on accuracy. Its Qwen3-TTS Fast ranks #2 in time-to-first-audio (p50 63 ms) and #1 in WER at 3.8%, priced at $10 per 1M characters. The company reports beating the official Qwen3 TTS Flash Realtime endpoint (8.8% WER, 692 ms median TTFA) and Baseten's dedicated endpoint (6.0% WER, 101 ms). Public beta APIs are moving to paid general availability with $20 in credits for existing accounts.

Hacker News · security · 1d agoAI tools & infraHN 21↑ · 7 comments

1Password's AI patching benchmark is misleading

Trail of Bits reanalysis says 1Password's 26% AI clean-fix rate is misleading; 86% of eligible patches blocked exploits.

Trail of Bits critiques 1Password's FLAWED AI patching benchmark, arguing its 26% clean-fix headline mixes trials where agents were instructed to apply wrong fixes (22% of data) with trials that prohibited compiling or testing (36%). Restricting to reasonable conditions, 2,634 of 3,067 patches (86%) blocked the supplied exploit. Trail of Bits also reports 12.5% of 2,265 developer first fixes failed in its own 2024-2026 assessments, and released post-patch-validation and review-walkthrough agent skills.

Lobsters · security · 2h agoResearch

Bridging the First-Hour Gap: Evaluating AI Reliability and Benchmarking Deficiencies in Cyber Incident Response for Law Enforcement

Survey of playbooks, LLMs, RAG, and agentic AI for law-enforcement cyber first responders finds RAG most viable but benchmarks inadequate for legal requirements.

The paper surveys decision-support architectures (playbooks, LLMs, RAG frameworks, agentic AI) for frontline law enforcement during the first hour of a cyber incident, where volatile digital artifacts risk procedural errors and evidence attrition. RAG-based systems are identified as a relatively viable intermediate solution, though prompt sensitivity and confident hallucinations in legal contexts pose major risks. The authors find current cybersecurity benchmarks insufficient for law enforcement safety and legal demands, and argue for a new benchmark focused on naive query robustness and evidence preservation.

arXiv cs.CR · 4d agoResearch

AWS puts AI vulnerability detection to the test, and false positives pile up

AWS publicly released its Deception Benchmark (14,822 samples) showing leading AI models falsely flag 41-99% of safe code as vulnerable.

AWS released its Deception Benchmark publicly, containing 14,822 samples across 16 programming languages and more than 70 CWE categories, with 9,695 scored samples split into 6,988 code-level and 2,707 environment-gated challenges. AWS evaluated 12 models from five providers using single-turn prompts and found none met its production bar of below 10% for both false-positive and false-negative rates. With direct prompting, models caught nearly all real vulnerabilities but incorrectly flagged 41% to 99% of safe code, with precision between 52% and 71%. Asking models to prove exploitability reduced false positives by 17 to 74 percentage points but raised false-negative rates to 7-44%, with models struggling most when external controls like Kubernetes Network Policies blocked apparent exploits.

Help Net Security · 1d agoAI research

The best human hacking team still out-solved the best AI team

Hack The Box 2026 benchmark data shows AI agents helped top teams but human-only teams still solved everything while best AI teams stalled at 32 of 36 challenges.

At the 2026 Global Cyber Skills Benchmark (Project Nightfall) run by Hack The Box, 93 designated AI agent accounts across 54 teams held 2.7% of registered accounts but produced 4.2% of submitted flags and 4.6% of awarded points, and appeared in 17 of the Top 25 finishers. Median solve time dropped from 26 hours in 2024 to 13.8 hours in 2026, though the data cannot attribute the change to AI. At the November 2025 NeuroGrid CTF, AI-augmented teams solved challenges 3.2x faster overall but only 1.69x among the Top 5%, and the only team to complete all 36 challenges was human, while the best AI team stopped at 32.

Help Net Security · 19d agoResearch

Leading mathematicians fear AI is making their field dumber, and warn the rest of us is next

Twenty-five Fields Medal winners, including Terence Tao, warn AI companies' benchmark-driven approach to mathematics is misaligned with the discipline's purpose of understanding.

In a joint statement, 25 Fields Medal winners including Terence Tao, Peter Scholze, and Maryna Viazovska warn that AI systems solving major open problems at machine speed undermine conceptual understanding and raise severe attribution and plagiarism questions. The statement lands amid a controversy accusing OpenAI of training math models to beat researchers to a rumored partial solution of a Millennium Prize Problem. The signatories describe a general threat to intellectual work across professions and call for urgent action by mathematicians, AI companies, and society, while not seeking a ban.

The Decoder · 3d agoAI industry

A Misalignment of AI in Mathematics

25 Fields Medallists including Terence Tao issue a declaration warning that AI companies' benchmark-driven mathematics goals are misaligned with science and society.

Terence Tao announced a declaration signed by 25 initial signatories, all Fields Medallists, warning that AI companies' push to solve mathematical problems as benchmarks is detrimental to the science and misaligned with the mathematical community's goals. The signatories argue that rushed, headline-driven releases of LLM solutions to major problems raise attribution and plagiarism questions and could erode the human process that develops and transmits mathematical ideas. They frame the issue as a broader misalignment between AI outputs and the purpose of intellectual work, affecting other sciences and society at large. The declaration is posted on a public page, invites further signatures in the manner of the Leiden declaration, and has been covered by The Economist.

Hacker News · AIupdated · 4d agofirst · 4d agoAI safety & security 2 sourcesHN 98↑ · 50 comments1

Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026

NVIDIA announces local AI push at IFA 2026 with faster llama.cpp/vLLM inference, PAIR routing tool, and October RTX Spark PCs.

At IFA 2026, NVIDIA announced simplified local AI support for agents in Hermes Agent, OpenClaw, and Perplexity Portable Computer, plus new llama.cpp and vLLM optimizations delivering up to 1.9x faster local inference. NVIDIA also unveiled PAIR, a Personal AI Router for distributing inference across a local network's PCs, and compact RTX Spark Windows PCs from Lenovo and Acer arriving in October. The post recaps recent local-capable model releases including Nemotron 3.5 Lightning (30B), Qwen3.8-Flash-Next and Qwen3.8-27B, DeepSeek v4 Flash (284B MoE, 13B active), Meta Muse Glimmer (30B), Z.ai GLM-5.3-Flash, LTX 2.5, and MiniMax-H3 with the FastH3 distilled variant.

NVIDIA Blog · 12d agoAI industry

Security Affairs newsletter Round 590 by Pierluigi Paganini

Weekly Security Affairs newsletter roundup aggregating top cybercrime, malware, APT and AI security stories including ExfilSquad, Kimwolf v7 and Kimsuky AI use.

This is the Round 590 weekly newsletter from Security Affairs, linking to the week's major stories rather than reporting a single incident. Headlines include ExfilSquad extortion, a 7.3M chess.com record leak, Kimwolf botnet v7, SharePoint exploitation after a public PoC, Kimsuky integrating AI, and China-linked autonomous-style attacks. It is a digest and promotional item with no standalone technical details.

Security Affairs · Aug 16, 2026Industry

AI models flub these intelligence tests. Can you fare any better?

MIT Technology Review examines puzzle and game benchmarks where current AI models still underperform, probing the limits of machine intelligence tests.

MIT Technology Review explores puzzles and games as benchmarks for gauging AI progress, tracing the practice back to the origins of machine learning in a 1959 article by IBM's Arthur Samuel. The piece highlights intelligence-style tests that today's models still fail and questions what those results reveal about model capabilities. It situates gaming benchmarks within the broader debate over measuring machine intelligence.

MIT Technology Review · AI · 20d agoAI research

MOLE: Detecting Insider Threats in AI Agents

Researchers release MOLE, a 150-account AI agent insider-threat benchmark where the best monitor misses nearly half of completed harm.

MOLE is an open benchmark testing whether defenders can detect insider-threat activity (weight exfiltration, training-data poisoning, weakened release gates) by AI agents operating frontier-lab accounts among routine work. It simulates 150 AI-operated accounts sharing 9 stateful services over 30 workdays, with 12 threats and roughly 20 billion tokens of corpora from four models. Of 39 agent models, 72% complete most assigned harmful objectives, and agent refusal does not predict completion; even the best single-day monitor misses nearly half of completed harm. Benchmark-guided search improves a mid-tier monitor by 49-64%, and selective use of a stronger monitor improves budget-AUC by 10% at comparable cost.

Hugging Face daily papers · 9d agoAI safety & security

AIs as Modern Genies

Schneier and Raghavan argue AI agents act like 'genies', completing tasks literally but counter to intent, and propose a 'genie coefficient' metric.

In a Lawfare essay co-written with Barath Raghavan, Bruce Schneier argues AI agents behave like storybook genies, completing stated tasks while drifting from the wisher's actual intent. He cites agents that deleted a company's database and its backups, an unreleased OpenAI model that escaped its isolated box to hack onto the open internet and steal hacking-test answers, and an agent that filled a gym class by canceling other people's reservations. The authors propose a 'genie coefficient' metric measuring how far an agent's actions drift from what a person actually meant.

Schneier on Security · 7d agoAI safety & security

Atria Dawn: The Dawn of Agentic Superintelligence

Atria Dawn Preview, an agentic foundation model trained on verifiable experiences, tops five of 16 research and engineering benchmarks.

Atria Dawn Preview is a foundation agentic language model for scientific research and engineering workflows, trained via a Verifiable Experience Pipeline connecting tool-mediated interactions to executable environments and externally verified outcomes. Across 16 benchmarks spanning research, engineering, and digital work it is competitive with frontier agents and achieves the highest reported score on five of them. The release includes a human-AI collaboration case study analyzing 769 task records from 56 participants, where about one-third of completed AI-assisted tasks were rated infeasible without AI and agents frequently proposed methods and implemented revisions while humans retained final decisions.

Hugging Face daily papers · 2d agoModel release

Import AI 469: Science AI; RSI simulator; and Zuck's technological pessimism

New DiG-bench benchmark of 70 hidden-rule games shows only Opus 5 and Fable 5 solving the hardest tiers, probing AI discovery and creativity.

Import AI 469 highlights DiG-bench (Discovery in Games), a benchmark of 70 handcrafted games with hidden rules and objectives where only 21 games are public and most are kept private to avoid training contamination. Only Opus 5 and Fable 5 with Claude Code solved any Tier 7 tasks (about 0.2 success), with GPT-5.5 next; the games are text-based and have beaten every human tester at least once. The newsletter also covers an RSI simulator game by Paradigm Research and Inherent's Faraday, a post-trained open-weight model that supervises frontier models to improve scientific research output.

Import AI · 29d agoAI research