ZeroHour

Search: “machine-learning”

238 stories

Why don't machine learning research agents overfit?

Amazon researchers explain why ML research agents avoid benchmark overfitting, attributing generalization to compressibility of successful strategies.

Amazon Science summarizes the paper "What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents," which investigates why benchmark hill-climbing loops, whether run by human communities or LLM research agents, do not produce rampant overfitting. The explanation formalizes Occam's razor via a counting argument: successful ML strategies are highly compressible, so short descriptions lack room to memorize benchmark data and must capture real structure. LLM-based agents, being resettable and controllable, allow this hypothesis to be tested empirically.

When the prompt becomes the payload: A practical pen-testing guide for GenAI, LLM and RAG applications

CSO Online publishes a practical penetration-testing guide for GenAI, LLM, and RAG applications, covering prompt injection, retrieval poisoning, and tenant isolation testing.

The guide frames LLM applications as attack graphs spanning prompts, retrieval layers, vector stores, tools, identities, and downstream APIs, arguing that conventional web testing misses instruction-vs-data channel risks. It builds on OWASP prompt injection guidance (direct vs. indirect injection) and NIST's 2025 adversarial machine-learning taxonomy, noting that RAG and fine-tuning do not remove injection risk. Recommended practices include documenting trust transitions across components, using canaries and synthetic records to avoid test side effects, running multi-turn and obfuscated injection campaigns, and verifying chains from poisoned documents to observable state changes. It also details testing RAG pipelines via controlled document poisoning across metadata, OCR layers, and code comments, plus cross-tenant isolation checks on retrieved document IDs.

CSO Online · 6d agoAI safety & security1

DeepSeek, Alibaba and Chinese AI Firms Extract Billions of Tokens From U.S. AI Models

NSA, CISA and FBI advisory AA26-251A accuses DeepSeek, Alibaba and four other Chinese AI firms of industrial-scale distillation of US frontier models.

Joint advisory AA26-251A from NSA, CISA and FBI accuses DeepSeek, Alibaba, Moonshot AI, MiniMax, StepFun and Z.AI of extracting billions of tokens from Claude, GPT, Gemini and Grok variants since at least late 2024, likely with Chinese government awareness. Campaigns allegedly used API proxy 'transfer stations', account pools, metadata sanitization and prompt injection to harvest reasoning, coding, agentic and reinforcement-learning capabilities, with techniques mapped to MITRE ATLAS. DeepSeek's R1 and V3 and Alibaba's Qwen families reportedly trained on harvested outputs, and DeepSeek's $5.6 million training-cost claim is disputed as excluding distilled data value. Agencies urge anomaly monitoring, output alteration for suspected extractors, and intelligence sharing across vendors, clouds and aggregators.

GBHackersupdated · 4d agofirst · 6d agoAI safety & security in the wild 13 sources1· 1 read

Google’s AI weather model now uses more raw satellite data

Google launched WeatherNext 3, an AI weather model using raw satellite data that beats ECMWF and now powers Search, Gemini, and Maps.

Google released WeatherNext 3, an AI weather forecasting model that incorporates physical surface information (land/ocean type and elevation) to improve surface temperature and dewpoint calculations, improving point location temperature accuracy by up to 30 percent. Its white paper reports roughly 5 percent better upper-atmosphere accuracy than WeatherNext 2, equating to about six additional hours of forecast lead time, outperforming the ECMWF AI model on these metrics. The model now supplies forecast information across Google Search, Gemini, and Maps, though the paper notes unexplained short-lead degraded results and grid-shaped artifacts in some predictions.

Ars Technica · AI · 7d agoModel release

Attackers Expose Ongoing AI Tool Use Targeting Organizations in Latin America

Unit 42 documents two AI-assisted intrusion campaigns against Latin American government, utility, and financial organizations using LLM-orchestrated tooling.

Palo Alto Networks Unit 42 tracks two ongoing intrusion clusters, CL-CRI-1131 (Mexican transportation, federal ministries, municipal water utilities) and CL-CRI-1163 (Brazilian financial sector), both using living-off-the-land techniques, SOCKS5 relays, and custom RATs. The attackers appear to orchestrate operations via commercial LLMs like Claude and GPT-4.1, evidenced by iterative batch scripts and AI-generated tunneling tool naming. The Mexican campaign (also reported as Operation Escaneo by CloudSEK) exfiltrated sensitive data via dynamic-DNS infrastructure with rotated multi-SAN TLS certificates between February and June 2026. This signals broader adoption of AI-enhanced operations by diverse threat groups in the region.

Palo Alto Unit 42 · 12d agoThreat actor in the wild1

U.S. CISA adds a MLflow flaw to its Known Exploited Vulnerabilities catalog

CISA added actively exploited MLflow SSRF flaw CVE-2026-64849 (CVSS 9.3) to its KEV catalog; attackers are stealing cloud credentials from exposed instances.

CISA added CVE-2026-64849, a critical unauthenticated server-side request forgery in MLflow, to its Known Exploited Vulnerabilities catalog. The flaw affects MLflow versions before 3.15.0 and allows unauthenticated attackers to reach internal services including cloud metadata endpoints, exposing temporary cloud credentials. watchTowr observed in-the-wild exploitation exfiltrating credentials and secrets, plus widespread scanning of exposed MLflow instances within hours of the CVE's assignment on August 17, 2026. MLflow is an open-source platform for managing the machine learning and AI development lifecycle with over 60 million monthly downloads.

Security Affairs · 26d agoExploit / PoC in the wildCVE-2026-64849

Almost Half of Malware Samples Communicate Direct to IP

Unit 42 analysis of 4 million malware reports finds 45% of C2-active samples connect directly to hard-coded IPs, bypassing DNS defenses.

Palo Alto Unit 42 analyzed over 4 million Advanced WildFire dynamic analysis reports and found that 45.32% of malware samples with C2 activity made at least one direct-to-IP connection, accounting for 23.17% of all C2 connection attempts. The firm proposes zero trust IP (ZT-IP), an enforcement approach that verifies whether outbound destinations were ever sanctioned by a DNS response. ZT-IP analysis surfaced Phorpiex ransomware droppers fetching payloads directly from C2 IPs, a persistent data exfiltration campaign using an obfuscated \GET protocol, and Mozi P2P botnet payloads delivered to IoT devices without DNS. Only 1% of benign samples connected directly to untrusted IP addresses.

Palo Alto Unit 42 · 29d agoResearch