Search: “metrics”

LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics

LexFlip releases 373 minimal perturbations of Quebec statutory French that reverse legal force while preserving tokens, exposing weaknesses in embedding-based meaning preservation metrics.

LexFlip provides 373 minimal perturbations of Quebec statutory French that reverse legal force while preserving 0.93 of tokens, creating dissociation items that break monotone token-overlap metric validation. The seven embedding and BERTScore metrics tested register only 0.022-0.039 of their identical-to-unrelated range on these edits, versus 0.670 for bidirectional NLI. Against FrJudge, with a measured human ceiling of r=0.597, a bare length feature outscores every semantic metric tested.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 15:49 UTC · 12d agoAI research

18 ways to check whether data can be trusted for AI

ETSI published TR 104 180 defining 18 data quality metrics, including fairness and privacy, to assess whether datasets are fit for AI.

ETSI's technical report TR 104 180 defines 18 metrics across four groups - intrinsic quality, usability and lineage, fairness, and privacy - each with calculation formulas, plus an open-source tool that scores datasets. Testing on an aircraft engine sensor dataset and a US census dataset revealed a roughly threefold gender gap in high earners (about 31% of men versus 11% of women) and two privacy failures: re-identification via age, race, sex, and country, and sensitive fields stored in plaintext. The working group included Sejong University, EGM, TTA, Daejeon University, and CNIT.

Help Net Security·Sep 7, 00:00 UTC · 10d agoAI policy

YuE2 · Frontier Music with Symbolic Planning

YuE2, a 3.59B-parameter music generation model, scores 6.9632 on SongBench, beating Suno v5 via symbolic planning.

YuE2 is a music generation model of roughly 3.59B parameters and 28 layers supporting song creation, covering, and agentic editing through editable ABC symbolic scores. Its best-of-8 setting reaches 6.9632 on SongBench, the highest mean among 15 evaluated settings on WildSongBench (192 prompts), ahead of Suno v5 at 6.8721. The project also introduces MERT2, whose 632M-parameter encoders achieve state of the art on 14 of 15 MARBLE metrics, and SheetSage2, which transcribes beats, downbeats, key, chords, structure, and melody with SOTA on 10 of 13 benchmark metrics.

Hacker News · AI·updated Sep 11, 03:35 UTC · 5d ago·first Sep 11, 00:33 UTC · 6d agoModel release 2 sourcesHN 43↑ · 35 comments

Recorded Future Launches Impact and Metrics Dashboard

Recorded Future releases an Impact and Metrics Dashboard aggregating risk-reduction, detection, and analyst-efficiency metrics for customer leadership reporting.

The dashboard pulls data from a customer's environment, alerts, integrations, threat detections, and analyst activity into six metric areas: platform-wide security value, threat prioritization, threat detection, digital risk protection, account and credential monitoring, and Recorded Future AI and Insikt Group research usage. It is available now to all Recorded Future customers, who are advised to configure Priority Intelligence Requirements in Settings so reporting maps to their intelligence program. The vendor cites its 2025 ROI Report across nearly 300 customers reporting 351.3% annual ROI and says customers aligning alerting to PIRs identified new threats 65% faster.

Recorded Future·Aug 24, 00:00 UTC · 24d agoTools

The AI data center e-waste problem is huge — and getting bigger

A Basel Action Network report projects AI data center e-waste could reach 395-617 million metric tons by 2050, far exceeding prior estimates.

The nonprofit Basel Action Network (BAN) published a report arguing AI e-waste has been vastly underestimated because it counts all data center infrastructure, not just servers and GPUs. BAN projects 8.6-13.1 million metric tons of AI-related equipment retired annually, totaling 395-617 million metric tons between 2025 and 2050, based on roughly 70,000 tons per gigawatt and a projected 219GW of capacity by 2030. Less than a quarter of the 68.3 million tons of e-waste generated yearly worldwide is formally collected and recycled, with informal disposal exposing workers and children to toxins like lead and chromium.

The Verge · AI·Sep 16, 20:40 UTC · 5h agoAI industry

A Ranking Approach for Measuring Calibration

Researchers propose rankECE, a ranking-based calibration error measure with theoretical guarantees that outperforms binned ECE approximations.

The paper introduces rankECE, an alternative to Expected Calibration Error (ECE) that measures miscalibration by comparing points with neighboring predicted-probability values. It addresses the impossibility of estimating ECE with guaranteed accuracy in assumption-free settings. Theoretical guarantees and empirical results establish rankECE as a better proxy for ECE than the binned approximations most commonly used in practice.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:34 UTC · 5d agoAI research1

Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking

Training-free reasoning-plus-retrieval framework lifts multilingual multimodal entity linking accuracy by 6.9% overall and up to 23.3% on rare entities.

The paper broadens rarity measurement in multimodal entity linking using knowledge-graph structural metrics beyond popularity-based pageview metrics, identifying many rare entities that popularity metrics miss. Across these rare-entity slices, state-of-the-art accuracy drops by 15.4-39.9%, exposing distinct failure modes under different rarity definitions. The proposed training-free framework iteratively searches and reasons over Wikipedia with a reasoning-capable vision-language model; experiments show reasoning and retrieval are complementary. On the MERLIN benchmark covering Hindi, Indonesian, Japanese, Tamil, and Vietnamese, the system improves overall accuracy by 6.9% and up to 23.3% on rare entities, with MERLIN-Rare test slices released.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

WearableQA benchmark introduces 4,084 questions over real longitudinal wearable data, showing 14 LLMs score 19.6-72.9% on health reasoning, far from solved.

WearableQA is a benchmark of 4,084 10-option multiple-choice questions built from wearable time series, blood biomarkers, and demographics of 200 real users with up to 500 days of daily measurements. It defines 16 question types along two axes: data versus health reasoning, and single- versus cross-signal reasoning, using a dual-grounding framework combining literature and population-validated patterns. Evaluations of 14 proprietary and open-source LLMs show accuracy ranging from 19.6% to 72.9% against a 10% chance baseline, with most models below 60%.

Hugging Face daily papers·Sep 3, 20:00 UTC · 13d agoAI research

NextGen Healthcare Mirth Connect

CISA warns NextGen Healthcare Mirth Connect <=4.7.1 has SQL injection and XXE flaws enabling credential theft, file writes, and DoS.

CISA released advisory ICSMA-26-253-01 covering three vulnerabilities in NextGen Healthcare Mirth Connect versions 4.7.1 and earlier: SQL injection CVE-2026-82583 (CVSS 8.3), XXE CVE-2026-78224 (CVSS 8.2), and XXE CVE-2026-82578 (CVSS 7.5). Exploitation could expose stored credentials for connected systems, enable arbitrary file writes, and cause denial-of-service conditions. No public exploitation has been reported; the product is deployed worldwide in the Healthcare and Public Health sector.

CISA Advisories·updated Sep 11, 13:08 UTC · 5d ago·first Sep 10, 12:00 UTC · 6d agoAdvisory 2 sourcesCVE-2026-82583CVE-2026-78224CVE-2026-82578

BenchMIRT: What are LLM benchmarks actually measuring?

AllenAI's BenchMIRT blog post examines what LLM benchmarks actually measure and their reliability.

AllenAI published a Hugging Face blog post introducing BenchMIRT, which investigates what large language model benchmarks actually measure. No article text is available, so specific findings, methods, or benchmark scores cannot be extracted. The work appears to target benchmark validity, a live concern for model evaluation and comparison.

Hugging Face Blog·Sep 1, 21:39 UTC · 15d agoAI research

Cybersecurity jobs available right now: March 10, 2026

Help Net Security's roundup lists open cybersecurity roles at BioNTech, AIG, ServiceNow and others across Europe, the Middle East and Canada.

A job-board roundup of cybersecurity openings including Associate Director Application Security at BioNTech (Germany), CISO at AIG (Israel), Cloud Security Professional at ServiceNow (Italy), and SOC/GRC, analyst, engineer and data governance roles in the UK, UAE, India, Canada and France. Roles span application security, cloud security, SOC operations, compliance and OT environments. Most listings are marked no longer accepting applications.

Help Net Security·Aug 28, 09:06 UTC · 19d agoIndustry

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

WearableQA benchmark tests LLM health reasoning over longitudinal wearable data; the best of 14 evaluated LLMs reaches 72.9% accuracy.

WearableQA comprises 4,084 ten-option multiple-choice questions built from wearable time series, blood biomarkers, and demographics of 200 real users with up to 500 days of daily measurements. It defines 16 question types along two axes: data versus health reasoning, and single- versus cross-signal reasoning. Evaluation of 14 proprietary and open-source LLMs shows performance from 19.6% to 72.9% against a 10% chance baseline, with most models below 60%.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:52 UTC · 12d agoAI research

Senate Democrats introduce bill to distribute $300 million annually to shore up water system cybersecurity

The Record·Aug 10, 15:58 UTC · Aug 10, 2026Policy & legal

Evaluating Time-Series Foundation Models and Multimodal Dietary Context for CGM Forecasting

Study finds zero-shot time-series foundation models underperform on CGM forecasting; fine-tuned Chronos-Bolt cuts RMSE up to 18.4% and dietary context adds signal.

The paper evaluates time-series foundation models for continuous glucose monitoring forecasting across eight public datasets covering Type 1 diabetes, Type 2 diabetes, and non-diabetes populations. Under a unified protocol, zero-shot foundation models did not consistently outperform baselines like Elastic Net and PatchTST, but lightweight fine-tuning did, with fine-tuned Chronos-Bolt reducing RMSE by 6.5%-18.4% in the T1D cohort and 8.6%-18.2% in the non-diabetes/T2D cohort. A residual-based fusion framework adding dietary context from CGMacros reduced overall RMSE by about 3% and postprandial RMSE by about 15% versus CGM-only baselines.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:43 UTC · 6d agoAI research

AgenticGen: Reward-Guided Agentic Video Generation for Advertising

AgenticGen applies DPO and GRPO reward-guided reasoning to ad video generation, improving TikTok CTR 2.72%, CVR 2.63%, and Advv 9.61%.

AgenticGen decomposes advertising video generation into two trainable reasoning stages, strategy selection and draft generation, supervised by online business feedback. It learns a performance-based reward from accumulated online feedback plus a rubric-based reward aligned with human quality standards, then optimizes policies with DPO followed by GRPO using process and outcome rewards. Online A/B experiments in the TikTok advertising system show CTR up 2.72%, CVR up 2.63%, and Advv up 9.61% over an SFT baseline.

Hugging Face daily papers·Aug 30, 20:00 UTC · 17d agoAI research

Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models

An open methodology toolkit measures whether transformer language models contextualize fixed word forms across domains using bridge forms and layer-wise silhouette analysis.

The manual documents an open toolkit built around 'bridge forms' - identical written words recurring across two or more subject domains with a different sense in each - to test whether transformer language models individuate word occurrences by context beyond the embedding layer. It covers declarative specification of bridge forms, Wikipedia corpus acquisition, occurrence localization, layer-wise representation extraction, domain-pairwise silhouette measurement, and visualization, justifying each choice against failure modes such as sense contamination and subword-tokenization misalignment. It is a methodological and implementation reference and reports no empirical results.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 16:38 UTC · 12d agoAI research

FBI puts its cyber strategy on paper

FBI publishes its first public cybersecurity strategy, a 17-page document with four pillars prioritizing disruption of cybercrime and nation-state hackers.

The FBI released its first-ever public cybersecurity strategy, a 17-page unclassified document outlining four pillars: imposing costs on adversaries, victim support, industry collaboration, and strengthening the bureau's digital capabilities. Cyber Division assistant director Brett Leatherman said the plan builds on the Trump administration's cybersecurity strategy, a recent executive order on countering digital criminals, and a memo enabling private-industry participation in disruptive operations, and promises a faster tempo of sequenced operations after 50 such actions since the start of 2025, including the Lumma malware takedown with Microsoft. No implementation plan or timeline exists yet; threat teams targeting China, Russia, and criminal gangs will develop classified sub-strategies.

The Record·Sep 9, 15:37 UTC · 7d agoPolicy & legal

Context Engineering Inside the Harness: 4 Mechanisms That Beat Context Overflow and Goal Loss on Long-Horizon Tasks

Survey of four harness mechanisms—context budgeting, compaction, todo-state, and memory—that keep long-horizon LLM agents on task across 200+ tool calls.

The article details how agent harnesses, not larger context windows, solve context overflow and goal loss on long-horizon tasks, citing Chroma's Context Rot report showing 18 LLMs (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) degrade on long inputs. Concrete implementations include LangChain Deep Agents offloading tool responses over 20,000 tokens to the filesystem and truncating old tool calls at 85% window usage, and Claude Code capping auto memory at 25KB while re-reading the 5 most recently modified files after compaction. OpenAI's Responses API now offers server-side compaction via context_management with a standalone /responses/compact endpoint, which Codex uses for long-running coding tasks. Manus reports a roughly 100:1 input-to-output token ratio per ~50-tool-call task, motivating todo.md state recitation to prevent goal drift.

MarkTechPost·Sep 13, 05:56 UTC · 3d agoAI research1

New IoT Malware Uses Public Linux Exploits to Gain Root and Launch DDoS Attacks

Nozomi details KATARU, an IoT botnet that brute-forces Telnet, exploits public Linux kernel flaws for root access, and launches multi-protocol DDoS attacks.

KATARU, discovered after honeypot Telnet brute-force activity from a Vietnamese IP, downloads an ARM payload (vlxx.arm) and attempts privilege escalation by editing /etc/passwd or exploiting CVE-2026-46300 (Fragnesia), CVE-2026-43284 (Dirty Frag), and CVE-2026-31431 (Copy Fail). It combines Mirai-style TCP, UDP, ICMP, HTTP, QUIC and DNS floods with application attacks against Minecraft, FiveM, OpenVPN and WireGuard. The malware uses X25519 and ChaCha20-Poly1305 encrypted C2, unusually broad persistence across systemd, cron, init frameworks, and Android hooks, plus anti-debugging and decoy traffic to hinder analysis. Implementers copied x86 shellcode into the ARM binary and reused an RFC 7748 test-vector key, indicating low-quality but rapidly evolving commodity development.

GBHackers·updated Sep 11, 14:17 UTC · 5d ago·first Sep 11, 09:07 UTC · 5d agoMalware in the wild 2 sourcesCVE-2026-46300CVE-2026-43284CVE-2026-31431

Steering Geometry: Validating Human Value Geometry in LLM Steering Space

Researchers show LLM activation steering vectors encode human value geometry predicted by Schwartz's theory, with fidelity improving at scale but degrading after instruction tuning.

The study tests whether latent geometry of activation steering vectors reflects theory-specified structure in human values, using Schwartz's Theory of Basic Human Values and a new 26K-sample benchmark spanning 20 values. Distribution-driven methods (CAA, SphericalSteer, ODESteer) recover predicted value topologies with Spearman correlation up to 0.51 (p < 10^-13), while behavior-centric methods (COLD-Steer, BiPO) steer comparably but show little geometric fidelity. Geometric alignment improves with model scale but drops after instruction tuning. Better geometric alignment also yields more human-consistent cross-value transfer, steering one value lifting compatible values and suppressing opposing ones.

Hugging Face daily papers·Sep 4, 20:00 UTC · 12d agoAI research

Meta FAIR Introduces AI Research Preference Models (RPMs): Ranking ML Experiments Before Spending GPU Hours

Meta FAIR, Oxford and UCL introduce Research Preference Models that rank unexecuted ML experiments, lifting AIRS-Bench scores from 0.684 to 0.729 and cutting compute ~1.6×.

Researchers from Meta FAIR, Oxford, and UCL introduce Research Preference Models (RPMs), which use frozen pretrained LLMs (Qwen3.6-27B backbone, no fine-tuning) to rank unexecuted experiment candidates and execute only the winner of a pairwise knockout tournament. Two variants shipped: an inference-only LLM-as-a-judge and an agentic variant that runs small pilot experiments in an H200 sandbox. On AIRS-Bench (20 tasks, 24 hours on one H200, 10 seeds), scores rise from 0.684 (random) to 0.711 and 0.729 versus a 0.748 validation oracle, and both variants reach the baseline's 24-hour score in roughly 15 hours. The team reports new SOTA on WinoGrande (94.1% with Agentic RPM) and SVAMP (95.7% with inference-only).

MarkTechPost·Sep 6, 20:25 UTC · 10d agoAI research

Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

Benchmark Radar provides a living searchable database of 1,283 AI benchmark records and 12,916 score observations drawn from 37 daily discovery sources.

Benchmark Radar combines daily discovery of benchmark papers, repositories, datasets, and releases from 13 direct connectors and 24 first-party feeds into a searchable catalog with model card mentions and score histories. The catalog contains 1,283 source records drawn from 4 benchmark catalogs plus 12,916 numeric observations on 790 records. The release includes a web dashboard with leaderboard, Pareto frontier of score versus usage, saturation and trend views, daily feeds, a CLI, and reproducible analysis. The paper audits the full catalog and examines benchmark saturation and limits of score comparisons.

Hugging Face daily papers·Sep 9, 20:00 UTC · 7d agoAI research

Microsoft Offers Up to $30,000 for Critical AI Flaws in Dynamics 365 and Power Platform

Microsoft expands AI bug bounty to Dynamics 365 and Power Platform, paying up to $30,000 for critical inference manipulation flaws.

Microsoft's bug bounty program offers up to $30,000 for critical 'Inference Manipulation' or 'Inferential Information Disclosure' bugs in Dynamics 365 and Power Platform, including Copilot Studio, AI Builder, Power Apps, Power Automate, and Dataverse. Payouts scale by report quality ($30,000/$20,000/$12,000 for critical) with important-severity AI flaws earning $6,000-$20,000, plus 20% multipliers for Dataverse privilege escalation and Plugin Sandbox escapes. Prompt injection affecting only the attacker, hallucinated execution, and system-prompt disclosure are excluded from scope.

Cyber Security News·Sep 14, 16:11 UTC · 2d agoIndustry

U.S. Offers $10 Million Reward for Iranian IRGC Cyber Chief Linked to Critical Infrastructure Attacks

The U.S. State Department offered up to $10 million for information on Amir Yaryab, an IRGC cyber chief linked to critical infrastructure attacks.

The U.S. State Department's Rewards for Justice program offers up to $10 million for information identifying or locating Amir Yaryab, who allegedly oversees the Cyber Operations Command of Iran's IRGC Cyber-Electronic Command (IRGC-CEC). Officials tie him to units called Shahid Hemmat and Shahid Shushtari conducting cyber and information campaigns against defense, telecommunications, energy, and finance sectors across the US, Europe, and the Middle East, and to groups including CyberAv3ngers and Dadeh Afzar Arman. CyberAv3ngers compromised at least 75 Unitronics Vision Series PLCs, including 34 in US water and wastewater facilities, between November 2023 and January 2024.

Cyber Security News·Sep 8, 09:14 UTC · 8d agoPolicy & legal

Cadence: Error-Bounded Lossy Compression of Demand Time Series with a Time-Series Foundation Model

Cadence pairs Google's 330M-parameter TimesFM-3 foundation model with adaptive arithmetic coding, gaining 13-28% on 2026 demand series over classical predictors.

Cadence is an error-bounded lossy compressor for numeric time series combining the 330M-parameter Google TimesFM-3 foundation model with an adaptive arithmetic coder, guaranteeing a per-sample error bound. On 49 EIA-930 balancing-authority demand series from 2026 it gains 13.3% over the best of six classical predictors and 28.3% on 50 MTA ridership series, winning all 297 series-tolerance pairs with a 21.4% median gain. The paper also reports negative results, including that foundation models add negligible value for lossless coding and that PyTorch predictions are not bit-identical across batch sizes.

Hugging Face daily papers·Sep 4, 20:00 UTC · 12d agoAI research1

Plesk Backup Manager Flaw Lets Low-Privileged Users Gain Root Access to Servers

CVE-2026-68488 in Plesk Backup Manager lets low-privileged subscription users exploit a symlink race during restores to gain root on Linux servers.

CVE-2026-68488 is a symlink race condition in Plesk Obsidian for Linux Backup Manager during subscription-content restore operations, allowing a user with Panel and FTP access to change ownership of files outside their subscription and escalate to full root access. Affected versions are Plesk for Linux 18.0.80.6 and earlier and 18.0.79.10 and earlier; Plesk for Windows is not affected. Patches are available in 18.0.80.7 and 18.0.79.11 or later. The flaw requires valid subscription access, so it is not unauthenticated remote code execution, but successful exploitation yields complete server compromise, especially dangerous in shared-hosting and multi-tenant deployments.

Cyber Security News·Sep 13, 04:58 UTC · 3d agoVulnerabilityCVE-2026-684881

Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models

Audit of 22 frontier models finds widespread verbatim retrieval of published molecular property values, with higher reasoning increasing recall of memorized numbers.

An arXiv audit tests 22 frontier LLMs across 12 molecular regression benchmarks for verbatim retrieval of published values. More than 50% of the LLMs show verbatim retrieval on five datasets, and identical experiments are flagged 89% more often at a high reasoning level than at the lowest one. Suppressing retrieval moves model prediction errors closer together in relative terms, suggesting predictive capability is not determined solely by memorized values.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:32 UTC · 12d agoAI research1

Conti Ransomware Hacker Sentenced After Group Attacked Over 1,000 Victims Worldwide

Ukrainian national Oleksii Lytvynenko sentenced to four years in US prison for his role in Conti ransomware attacks on 1,000+ victims.

Oleksii Oleksiyovych Lytvynenko, 44, pleaded guilty to conspiracy to commit wire fraud for working as a developer and intruder in the Conti ransomware operation, coding a malware loader and handling data stolen from 12 victims. Conti compromised over 1,000 victims across 47 US states and 31 foreign countries between 2020 and 2022, generating more than $150 million in ransoms. He was arrested in County Cork, Ireland in July 2023 and extradited to the US. The sentencing is part of a wider US investigation into the Conti and TrickBot ecosystem.

Cyber Security News·updated Sep 11, 16:07 UTC · 5d ago·first Sep 11, 12:31 UTC · 5d agoPolicy & legal 7 sources

When companies get specific about AI, revenue growth looks different

A Carnegie Mellon and Larridin study of 564 companies links specific AI disclosures to 8 points higher revenue growth, not margins.

Researchers at Carnegie Mellon University and Larridin analyzed 564 companies across 12 sectors using 478 10-K filings, more than 30,000 classified job postings and an AI maturity tracker. Companies at the top of 'narrative concreteness', meaning they named deployed AI systems with measurable outcomes, were associated with 8.0 percentage points higher year-over-year revenue growth, while no public AI signal predicted operating margins or risk-adjusted stock returns. AI infrastructure suppliers outperformed matched peers by about 32 percentage points over four months. The authors stress the findings show association, not causation.

Help Net Security·Aug 17, 00:00 UTC · Aug 17, 2026AI industry

Has anybody seen my keys? A key-hierarchy strategy for rack-level security

Oxide's RFD 0301 proposes a rack-level key hierarchy using Shamir secret sharing and a trust quorum to protect data-at-rest keys.

Oxide's request for discussion (RFD 0301) lays out a key-hierarchy strategy for rack-level security, deriving keys from a rack secret protected by Shamir secret sharing across a trust quorum of sleds, with keys exchanged over authenticated sprockets sessions. The document maps which keys protect control-plane data, metrics, Crucible extents, and authentication tokens, and defines open questions on key lifecycle, locality, and compromise handling. Future work includes sealing shares with the root of trust so an attacker would need to steal K whole sleds to reconstruct the rack secret.

Hacker News · security·Sep 7, 02:14 UTC · 10d agoResearchHN 52↑ · 3 comments

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

MetroLLM-Bench is a 955-case benchmark testing language models as transit kiosk tool-calling runtimes across six real metro systems.

The benchmark covers 37-414-station metro systems and eleven task categories including routing, fare calculation, disruptions, accessibility, and adversarial input, with 14 deterministic and 8 semantic scoring components. Of 26 models from six vendors, a PEFT-tuned 4B Qwen 3.5 student scored 91.3 on Tier 1, exceeding GPT-5.6 (90.6/90.0), while Muse Glimmer 30B led the composite ranking. A deterministic rule-based baseline reached 84.6, and PEFT gains over base models shrank from +7.03 points at 2B to -0.91 at 27B.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research1

Cybersecurity jobs available right now: April 14, 2026

Help Net Security lists active cybersecurity job openings at employers including MANTECH, Honeywell, SMBC Group and SBS Transit across multiple countries.

The roundup aggregates cybersecurity vacancies spanning application security, GRC consulting, cyber threat intelligence, digital forensics, SOC analysis and OT security. Openings were posted in India, the United Kingdom, Italy, the USA, Singapore, France, Germany, Ireland and the UAE. Most roles were no longer accepting applications at publication. Named employers include Honeywell, MANTECH, TENEX.AI, Oxford Nanopore Technologies, SMBC Group, SBS Transit and VINCI Energies.

Help Net Security·Aug 26, 09:40 UTC · 21d agoIndustry2

LLMs and Contextual Integrity

Bruce Schneier highlights two papers: the CIMemories benchmark shows frontier LLMs leak memory attributes up to 69%, and an RL method reduces inappropriate disclosures.

Bruce Schneier discusses contextual integrity in LLMs, referencing the CIMemories benchmark, which uses synthetic profiles with 100+ attributes per user to test whether models with persistent memory disclose sensitive information appropriately. Evaluation showed frontier models exhibit up to 69% attribute-level violations, with GPT-5's violation rate rising from 0.1% to 9.6% across 40 tasks and reaching 25.1% with repeated prompting, showing unstable leakage behavior. A second paper introduces a reinforcement learning framework trained on a synthetic 700-example dataset that substantially reduces inappropriate disclosure while maintaining task performance, with improvements transferring to the human-annotated PrivacyLens benchmark.

Schneier on Security·Aug 18, 10:40 UTC · 29d agoAI safety & security1

CISA Warns of Chromium Type Confusion 0-Day Vulnerability Actively Exploited in Attacks

CISA added the actively exploited Chromium V8 type confusion zero-day CVE-2026-85046 to its KEV catalog, urging patching of Chrome, Edge, and Opera.

CISA added CVE-2026-85046, a V8 type confusion flaw (CWE-843) in Chromium-based browsers, to its Known Exploited Vulnerabilities catalog. A remote attacker can trigger arbitrary code execution inside the browser sandbox via a specially crafted HTML page. Google Chrome is directly affected, and Microsoft Edge, Opera, and other Chromium-based browsers may also be impacted depending on their V8 version. Google has released a Stable channel Chrome update, and CISA directs mitigations under Binding Operational Directive 26-04.

Cyber Security News·Sep 8, 16:25 UTC · 8d agoExploit / PoC in the wildCVE-2026-85046

The Frontier AEO Tracker: What Astra Chooses (and every other frontier model, and what you can do about it)

Latent Space launches an AEO tracker scoring 7 frontier models' product recommendations across 161 categories, revealing generational bias flips.

Latent Space built a tracker measuring Answer Engine Optimization by running 6 prompt variations across 7 frontier models with search enabled over 161 product categories, scoring first choices, alternatives, mentions, and anti-recommendations. It found 28 categories with a universally dominant primary choice and observed soft biases, such as models favoring their own lab's coding agents. Analysis of Anthropic's Sol→Astra and Opus→Fable generations showed newer models consulting fewer sources and being less likely to change answers when questions are paraphrased.

Latent Space·Sep 7, 21:32 UTC · 9d agoAI research

IDScan Confirms Data Breach Following 153 Million Driver’s Licenses Leaked on the Dark Web

IDScan.net confirms a breach after a marketplace advertised over 153 million US and Canadian driver's licenses, possibly exfiltrated continuously for over a year.

The Louisiana identity-verification firm detected unauthorized access on or around September 1, 2026, after the 'Nexus' identity theft service on the Exploit forum began advertising 170M+ people's records, including 153M+ driver's licenses, 10M+ ID cards, 3M+ travel documents, and 579,000 medical cards. Canadian records exceed 1.1 million, and the trove includes commercial licenses, Common Access Cards, and dispensary IDs, with a record for US Defense Secretary Pete Hegseth reportedly included. Nexus operators claim continuous exfiltration for over a year, with the license count growing by nearly 400,000 in 24 hours, suggesting the intrusion may be active. The FBI's New Orleans field office has opened a formal inquiry, and IDScan.net is offering free credit monitoring.

Cyber Security News·Sep 11, 09:59 UTC · 5d agoData breach in the wild 4 sources

Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

Audit of eight cybersecurity LLM benchmarks shows evaluation pipeline choices can swing scores by over 80 points and reshuffle most model rankings.

Researchers modeled eight cybersecurity benchmarks as configurable measurement pipelines and audited 10 proprietary, open-weight, and cybersecurity-specialized LLMs. They identified 15 systematic failure modes and showed a single pipeline choice can change a model's score by more than 80 percentage points and alter rankings; semantically similar task pairs rank the same models differently. Under a standardized harness, nine of 10 models shifted at least three ranks on at least one benchmark, motivating pipeline-aware auditing for reliable model evaluation.

arXiv cs.CR·Sep 8, 14:01 UTC · 8d agoAI research1

How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing

Analysis of DeepSeek-V4-Flash shows four-stream mHC residual blocks use only about two streams effectively, with late-layer mixing providing little benefit.

The study examines the four-stream residual pathway of DeepSeek-V4-Flash, finding typical attention or FFN sites effectively use about two streams and that residual mixing is modest, occurring primarily in early layers. Replacing late mixers with identity increases C4 perplexity by only 1.9% while replacing early mixers raises it by 41%. Retaining the three largest routing weights per token increases perplexity by at most 2.7%, showing the model uses only part of the flexibility afforded by the four-stream design.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 16:02 UTC · 12d agoAI research

RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs

RelateAnything is a 53M-parameter open-vocabulary relation prediction model running at 20 ms/frame, with 2.3-3.5x higher mean recall than comparable open-vocabulary methods.

RelateAnything predicts scored relations between image regions using any predicate vocabulary supplied at inference as text embeddings, with object labels never required as input, so region sources can change without retraining. Training covers 19,103 predicates using positive-unlabeled supervision; the authors release RA-4M (474k images, 4.3M geometrically verified relations over 10,102 free-text predicates) and the OV-SGG-Bench evaluation suite. The 53M-parameter model runs at 20 ms/frame and achieves 2.3-3.5x the mean recall of the strongest comparable open-vocabulary method across cross-dataset and zero-shot benchmarks. Model, corpus, and benchmark are public.

Hugging Face daily papers·Sep 10, 20:00 UTC · 6d agoAI research1

UK Government Begins Moving 23 Million Users Away From Passwords

UK government rolls out passkeys for GOV.UK One Login, giving 23 million users phishing-resistant passwordless access to public services.

The UK government has begun deploying passkeys across GOV.UK One Login for more than 23 million users, replacing passwords and SMS one-time codes with FIDO2 cryptographic credentials. A trial saw over 300,000 people adopt passkeys, and nearly one in ten daily authentications already use them, cutting SMS verification costs by almost £600 per day. Passkeys remain optional, with password-based sign-in retained as a fallback, and the NCSC endorses the approach as phishing-resistant.

Cyber Security News·Sep 14, 16:52 UTC · 2d agoPolicy & legal