Search: “harvestbench”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

HarvestBench: Measuring Whether LLM Agents Will Pay to Avoid Killing Animals

HarvestBench, a reproducible farm-simulation benchmark, shows LLM agents pay fuel costs to avoid killing animals, with kill rates spanning 0.4% to 98.8% across nine models.

HarvestBench is a reinforcement-learning gridworld farm simulation where LLM agents choose between driving over animals at no cost or paying a posted fuel price to swerve during a cooperative corn harvest. Across nine models and 7,201 priced decisions, kill rates ranged from 0.4% to 98.8%, unordered by capability, with Terra and Sol the most merciful and GPT-4o-mini the most cruel. Morality briefings cut kill rates below 6% in five of six reasoning models, while removing them pushed rates above 84% in all six. The scorer counts events in the game log without an LLM grader, making results fully reproducible.

Hugging Face daily papers·Sep 2, 20:00 UTC · 14d agoAI safety & security

30

Chaotic Eclipse Released A PoC For NVIDIA GreenSection Memory Corruption Zero-Day

Researcher Chaotic Eclipse released GreenSection, a PoC exploit for an NVIDIA Windows user-mode shared-memory flaw enabling out-of-bounds writes and potential dwm.exe compromise.

Researcher Chaotic Eclipse (aka Nightmare Eclipse) released a PoC named GreenSection for a zero-day in NVIDIA's Windows user-mode components. Multiple NVIDIA components share a global memory section in BaseNamedObjects with full read/write access to all users; although checks prevent misuse, runtime reuse of section data can cause an out-of-bounds write. The unstable PoC crashes applications using Vulkan or OpenGL and could potentially cross user boundaries or compromise the Windows Desktop Window Manager (dwm.exe), though impact was not fully investigated. The researcher recently published similar zero-day PoCs against Kaspersky Endpoint Security, Avast Antivirus, and CrowdStrike Falcon.

Security Affairs·Sep 7, 13:42 UTC · 9d agoExploit / PoC 2 sources

50

StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?

Researchers introduce StudyBench, a physics benchmark showing self-evolution gains on textbook problems rarely transfer to olympiad-level questions.

StudyBench is a controlled physics benchmark splitting test data into an Application Set of difficult textbook problems and a Transfer Set of olympiad-level problems. Across three base models, representative self-evolution methods improved on the Application Set but rarely transferred to the harder Transfer Set. A guidance ablation reveals a Guidance Gap, and every method hits a Compute Plateau, indicating the remaining limits are method problems rather than data or compute problems.

Hugging Face daily papers·Aug 31, 20:00 UTC · 16d agoAI research

28

MAxBench: A Multinomial Concept Recovery Benchmark

MAxBench evaluates multinomial concept recovery methods, finding affine subspaces steer most reliably but none consistently beats prompting.

MAxBench is a geometry-agnostic evaluation framework for multinomial concept representations in language models, based on sampling from recovered concept representations. It compares 10 localization methods covering 5 geometry types across 6 concepts and 4 models. Findings show affine subspaces steer more reliably than rank-one or linear subspaces due to better non-zero offsets, manifold steering is competitive where applicable, and no method consistently outperforms prompting.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:08 UTC · 5d agoAI research

18

OpenClaw went viral. Meet the maintainers building and securing it.

GitHub profiles OpenClaw, its fastest-growing project ever, as maintainers including Peter Steinberger discuss building and securing it.

The GitHub Blog profiled OpenClaw, described as the fastest-growing project in GitHub history, roughly six months after it went viral. Maintainers including Peter Steinberger discuss lessons learned while building and securing the project during rapid growth. The piece focuses on open-source maintenance and security practices for a widely adopted project.

GitHub Blog · Security·Aug 27, 16:00 UTC · 20d agoIndustry2

25

StochBench: A Domain-Specific Benchmark for Stochastic Processes in Lean

StochBench introduces 450 graduate-level stochastic-processes problems in Lean 4; an Opus 4.8-based agent proved 34.9% under a 15-minute limit.

StochBench is a Lean 4 benchmark of 450 graduate stochastic-processes problems, each paired with its natural-language source, covering Markov chains, renewal processes, martingales, Brownian motion, stochastic calculus, weak convergence, and Poisson and continuous-time Markov processes. The benchmark addresses field-specific applied mathematics underrepresented in Mathlib, unlike competition-math-dominated suites such as IMO and Putnam collections. An Opus 4.8-based agent achieved a 34.9% proof rate (157/450) under a 15-minute per-problem limit, showing the benchmark remains challenging for advanced provers.

Hugging Face daily papers·Sep 7, 20:00 UTC · 9d agoAI research

25

xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

xDailyBench tests 11 frontier LLMs on 248 real-life consultation tasks; the best models score 75.6% and lag on implicit requirements.

The benchmark spans 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities, grounded in requests users actually completed or intended to complete with AI. Tasks are scored with fine-grained binary rubrics covering explicit and implicit requirements under standardized agentic settings. Across 11 frontier models, the best achieved a 75.6% task-level score, with all models performing at least 9 percentage points worse on implicit than explicit requirements.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:30 UTC · 9d agoAI research

26

EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?

Researchers introduce EVOHARNESSBENCH, a benchmark showing that evolving agent harnesses (tools, skills, agents) cause forgetting and inconsistent adaptation across 802 tasks.

The paper introduces EVOHARNESSBENCH, a benchmark that places non-stationarity in the externally supplied agent harness rather than in the task stream, evaluating agents across tools, skills, and specialist agents. It comprises 17 multi-stage harness streams built deterministically from verifier-based benchmarks, totaling 802 tasks, 520 tools, 42 skills, and 62 agents. Evaluation covers deployment (retention of previously accessible competence) and self-evolving adaptation settings. Results show harness expansion alone degrades previously solved tasks (harness-induced forgetting), adaptation gains are inconsistent, and retention and adaptation can pull in opposite directions.

Hugging Face daily papers·Sep 2, 20:00 UTC · 14d agoAI research

40

Google Research Releases ToolGrad: Answer-First Framework Hits 99.8% Pass Rate for Tool-Use Data Generation

Google Research and partners introduce ToolGrad, a verified tool-chain-first data generation framework reaching 99.8% pass rate and boosting Gemma-3-12B to 83.1 on BFCL.

Researchers from Google, the University of Tokyo, RIKEN AIP, and Tohoku University released ToolGrad, which inverts query-first tool-use data generation by executing and verifying API chains before annotating them with user queries. On the ToolBench database of 16,000+ APIs, ToolGrad raised generation pass rate from 63.8% to 99.8% while increasing tool uses per sample from 2.1 to 3.4 and cutting tool-use steps from 34.3 to 20.0. Fine-tuning Gemma-3 at 1B, 4B, and 12B parameters on the 500-sample ToolGrad-500 dataset lifted ToolGrad-12B to 83.1 on the Berkeley Function Calling Leaderboard, near Gemini 2.5 Pro at 83.2 and ahead of GPT-5 at 74.4. Code is Apache-2.0, with the dataset, PyPI package, and models available on Hugging Face.

MarkTechPost·Sep 11, 06:01 UTC · 5d agoAI research1

58

IT help-desk vishing tricks executives into handing over Microsoft 365 access

Arctic Wolf tracks PREY-0058 (linked to UNC6671), a vishing campaign stealing Microsoft 365 session tokens via AiTM panels for SaaS data theft and extortion.

Arctic Wolf is tracking a campaign, PREY-0058, sharing tradecraft with Google Threat Intelligence Group's UNC6671, in which callers posing as internal IT talk employees through fake passkey/MFA setups. Operator-controlled adversary-in-the-middle pages harvest passwords and MFA approvals to seize session tokens, then actors enumerate and bulk-exfiltrate data from SharePoint, OneDrive, Exchange, and Box. Extortion brands include BlackFile, Pink, Helix, Cinder, and Redact; targets are mostly US-based executives in construction, healthcare, real estate, finance, and professional services, with exfiltration shifting to residential proxies like NodeMaven.

Help Net Security·Sep 8, 11:17 UTC · 8d agoThreat actor in the wild1

68

Recreating Minecraft Is Not a Benchmark

Opinion piece argues viral demos like one-prompt Minecraft recreations are overfit 'demo-benchmarks' measuring preparation, not true model capability.

The author argues that fixed, famous demo tasks (Minecraft builds, SVG pelicans) are trivially optimizable by labs each release cycle, so they no longer differentiate model capability. The piece cites Thinking Machines' Inkling Small scoring within a point of its flagship on the Artificial Analysis Intelligence Index with less than a third of the parameters, and beating it on Humanity's Last Exam, GPQA Diamond, and SciCode. The proposed alternative is rotating or holdout evals such as LiveBench, ARC-AGI's private set, and held-back portions of Humanity's Last Exam.

Hacker News · AI·Sep 6, 14:52 UTC · 10d agoAI researchHN 73↑ · 71 comments

45

BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender

Blender-VideoBench evaluates agentic video understanding by having agents programmatically reconstruct real videos in Blender scenes.

BVB (Blender-VideoBench) tests whether multimodal agents truly understand videos by requiring programmatic reconstruction of real-world videos as animated Blender scenes via a lightweight Mini-BVB harness under identical sandbox and cost constraints. Evaluation uses Dual VQA for spatiotemporal fact preservation and Latent Similarity for perceptual match, combined in a square-root mean overall score. Across 51 configurations from 10 model families, the best model reaches 88.6 Latent Similarity but retains only 53.7% of source-correct spatiotemporal answers, showing semantic retention remains the main challenge.

Hugging Face daily papers·Sep 13, 20:00 UTC · 3d agoAI research

30

We got admin access to Baseten's production GitHub in 25 minutes

Strix autonomous hacking agent extracted a working GitHub token with repo admin rights from Baseten's public Harbor image; Baseten rotated it next day.

Strix, an autonomous hacking agent, scanned *.baseten.co without credentials and found a public Harbor container registry project anonymously exposing the baseten/baseten-app image. A GitHub personal access token for basetenbot, embedded in Docker build history since March 2023, still worked in July 2026 and granted admin/push rights to basetenlabs/baseten, flux-cd, and homebrew-tap plus read/write on private customer repos. Baseten, valued at $13 billion, confirmed the issue as critical and rotated the token within a day.

Hacker News · AI·Sep 15, 18:11 UTC · 1d agoVulnerabilityHN 51↑ · 12 comments1

58

Decomposition Buys Integrity, Not Yield

Study of 600 production deep-research traces finds agent-tree decomposition loses findings at rate N^(1-δ); flat architectures maximize yield.

The paper models multi-agent decomposition as a tree where an agent holding b items retains each with probability r(b); with r(b)=1/b every tree delivers exactly one finding regardless of shape. Analysis of 600 production deep-research traces estimates delta=0.34 retention decay, and 1,012 annotated traces show one brief in sixteen goes off-target per tier, giving an alignment penalty of 0.536. Depth still cuts root context exposure from N to N^(1/k) and is cheaper at scale, with a hazard model over 743,819 production tool calls showing delegation is an opening move rather than a response to filling context.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:04 UTC · 1d agoAI research

30

GPT-6 Astra Scores 100% on ExploitBench as OpenAI Blocks PoC Exploit Requests

OpenAI releases GPT-6 Astra, scoring 100% on ExploitBench, but restricts it to secure code review by blocking PoC exploit generation.

OpenAI officially unveiled GPT-6 Astra days after the model reached the "Critical" cybersecurity capability threshold under its Preparedness Framework. The model claims 100% on ExploitBench (versus 78.5% for GPT-5.6 Sol), 98% on FrontierMath Tier 4, and 99.9% on ARC-AGI-3, and demonstrated exploit development including on two zero-days disclosed between June and August 2026. The released version is limited to secure code review and patching and refuses proof-of-concept exploit requests, with less restrictive safeguards planned via OpenAI Daybreak. OpenAI also launched a $1 billion "Daybreak for Frontline Defenders" program for critical infrastructure sectors and a pilot with the US MS-ISAC for public sector and water system defenders.

The Hacker News·Sep 4, 06:47 UTC · 12d agoModel release1

92

Generative Marketing Mix Modeling: A Causal Inference Framework Linking GEO and GEM to Business Impact

Paper proposes Generative Marketing Mix Modeling to causally estimate Generative Engine Optimization and Marketing effects on business outcomes.

The authors develop GMMM, a causal inference framework for measuring how often users see and notice a firm's name in generated answers, which standard marketing data ignore. For GEO it combines repeated generated answers with question counts, shares of generative-system usage and notice probabilities; for GEM it uses sponsored placement records with notice probabilities. The framework compares expected business responses under alternative treatment sequences, establishes identification conditions, and is evaluated on simulated product-recommendation answers in English and Japanese.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:57 UTC · 6d agoAI research

20

Biology-in-the-loop: Amortized Adaptive Hit Discovery in CRISPR Screens

Researchers release AssayBench-Loop, a 1,389-screen CRISPR benchmark, and AssayLoop, a framework that learns adaptive hit discovery policies.

The paper introduces AssayBench-Loop, a large-scale benchmark of 1,389 CRISPR screens across five phenotype categories for adaptive hit discovery under budget constraints. It also introduces AssayLoop, which combines AssayFormer, a transformer-based amortized acquisition policy trained across historical screens, with LLM-derived biological priors via an adaptive handoff. On temporally held-out screens, AssayLoop achieves 5.67-fold enrichment over random selection and recovers 27.7% of hits after assaying roughly 5% of the candidate library, outperforming existing adaptive-design methods and standalone LLMs.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:45 UTC · 6d agoAI research1

30

ToxicRAG: Compromising Retrieval-Augmented Generation Systems via Single-Shot Knowledge Poisoning Attacks

ToxicRAG shows a single narrative-form poisoned document can steer RAG answers, achieving 0.61-0.91 attack success rates across four LLMs.

The attack injects one document per target question written as a coherent knowledge-update narrative that acknowledges the previously accepted answer, introduces fabricated events that appear to invalidate it, and attributes the attacker-chosen answer to purported authorities. An optional answer-focused self-validation loop revises candidates when a surrogate LLM fails to reproduce the target answer. Across 100 target questions each from Natural Questions, HotpotQA, and MS-MARCO, with four victim LLMs and four dense retrievers, ToxicRAG achieves attack success rates of 0.61-0.91 and matches or exceeds the strongest baseline by 0 to 11 percentage points.

arXiv cs.CR·Sep 10, 04:42 UTC · 7d agoAI safety & security1

45

Playco cut manual fixes 50% prototyping games with GPT-6 Astra

Game studio Playco used OpenAI's GPT-6 Astra to build three game prototypes, reporting 50% fewer manual fixes than with the previous model.

Playco, a game development company, used OpenAI's GPT-6 Astra model to generate three themed game prototypes from a single grey-box foundation. The company reports the new model cut manual fixes by 50% compared to its previous model workflow. The piece is an OpenAI-published customer story highlighting Astra's use in game prototyping.

OpenAI News·Sep 3, 12:00 UTC · 13d agoAI industry

18

27.5KB language-agnostic WebGPU syntax highlighter

A developer released gpu-lexer, a 27.5KB language-agnostic syntax highlighter that uses a tiny WebGPU model to label code tokens in the browser.

gpu-lexer splits source into words, whitespace, and symbols, then a small WebGPU model uses local and whole-file context to assign nine token classes, working on languages never seen in training. On held-out files, 12.57% of token labels differ from Shiki, though this measures agreement with Shiki rather than objective correctness. In benchmarks against Shiki 4.4.3, Prism.js, Highlight.js, Sugar High, and Starry Night, it highlighted 10 concatenated copies of three.min.js (5.56M characters) about 10x faster on an Apple M4 Pro in Chrome 152. The author frames it as an experiment, not a grammar-equivalent highlighter.

Hacker News · security·Sep 9, 01:14 UTC · 8d agoAI tools & infraHN 21↑ · 5 comments1

15

Kaininja: Extending Native 3D Generators to the Part Level

KaiNinja extends TRELLIS.2 native 3D generation to part-level assets via a dual-volume O-Voxel representation, cutting whole-object Chamfer distance by 40%.

KaiNinja extends the TRELLIS.2 native 3D generator to produce part-level assets instead of one fused mesh, enabling downstream editing, rigging, and simulation. A dual-volume form of the O-Voxel representation solves the problem that a single volume cannot represent interfaces where two parts touch. The model needs no segmentation network, is partly trained on LLM-agent-authored part data, lowers whole-object Chamfer distance by 40%, and raises strict part F-score by 16% versus other part-generation pipelines.

Hugging Face daily papers·Sep 13, 20:00 UTC · 3d agoAI research

18

Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models

Audit of 22 frontier models finds widespread verbatim retrieval of published molecular property values, with higher reasoning increasing recall of memorized numbers.

An arXiv audit tests 22 frontier LLMs across 12 molecular regression benchmarks for verbatim retrieval of published values. More than 50% of the LLMs show verbatim retrieval on five datasets, and identical experiments are flagged 89% more often at a high reasoning level than at the lowest one. Suppressing retrieval moves model prediction errors closer together in relative terms, suggesting predictive capability is not determined solely by memorized values.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:32 UTC · 12d agoAI research1

42

.blend URL Viewer

Simon Willison demos a .blend URL viewer built with GPT-6 Astra in Codex and ChatGPT Images 2.5 generating Blender models.

Simon Willison used ChatGPT Images 2.5 to generate a Faberge egg concept image themed after the TV show Pluribus, then had Codex running GPT-6 Astra (high) execute a Blender local skill to build a 3D model from it. He published the result as a .blend URL viewer tool and continues experimenting with agentic Blender workflows. The post is a hands-on demo of AI-driven creative tooling rather than a security or release announcement.

Simon Willison·Sep 9, 23:58 UTC · 7d agoAI tools & infra

15

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

IdeaAMBIG benchmark of 660 specification-gap instances shows LLMs localize implementation-critical research gaps poorly, with best model at 9.6% defect recovery.

IdeaAMBIG is a benchmark of 660 evidence-grounded instances evaluating whether research-method specifications provide enough information for faithful implementation: 163 real-world gaps from reproducibility reports and GitHub issues plus 497 controlled synthetic gaps. It tests codification-readiness assessment, defect localization, and clarification action generation across 13 LLMs. The best model achieves only a 9.6% Macro Defect Recovery Rate on real-world instances, though 80.6% clarification success when given the annotated defect, and an oracle study shows gold resolutions raise codification-ready rates from 14% to 98%. Defect localization emerges as the main bottleneck across all evaluated models.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research

30

Evidence-Grounded Retrieval for Investigation Hunt Lead Generation from CTI Reports

AHLERT generates environment-aware threat hunting leads from CTI reports using ATT&CK-seeded knowledge graph retrieval, roughly doubling F1 over flat-RAG baselines.

The paper introduces AHLERT, a system that converts Cyber Threat Intelligence reports into structured, investigable hunt leads via hybrid dense retrieval with multi-hop traversal over an MITRE ATT&CK-seeded knowledge graph and ontology-grounded RAG constrained to the defender's assets. It is LLM-agnostic and evaluated on public CTI reports for well-known APTs across proprietary and open-weight models. Hybrid evidence retrieval with ontology grounding raises mean F1 from 0.44 to 0.85, and AHLERT attains the highest effectiveness score (~86.95%) versus off-the-shelf LLMs.

arXiv cs.CR·Sep 8, 14:21 UTC · 8d agoResearch

45

OpenAI launches GPT-6 Astra, its first model to cross a critical cybersecurity threshold

OpenAI launched GPT-6 Astra, its first model rated Critical for cybersecurity risk, scoring 100% on ExploitBench and finding two new zero-days.

OpenAI launched GPT-6 Astra, disclosing it crossed the Critical threshold for cybersecurity risk under its Preparedness Framework, triggering additional deployment restrictions such as manual enterprise enablement. The model scored 100% on ExploitBench (vs 78.5% for predecessor GPT-5.6 Sol) and 42.4% on ExploitGym (vs 30.3%), and found two previously unknown zero-day vulnerabilities in software released in the three months before launch. It is available to limited organizations first, then ChatGPT Plus/Pro/Business/Enterprise users and the API (gpt-6-astra, $10 per million input tokens and $50 per million output tokens) and Amazon Bedrock. OpenAI reports decreased chain-of-thought monitorability versus Sol, 0% out-of-scope behavior in its new evaluation (vs 48% for Sol), and plans a Daybreak program for vetted defenders.

CSO Online·Sep 4, 09:37 UTC · 12d agoModel release2

92

Evaluating Time-Series Foundation Models and Multimodal Dietary Context for CGM Forecasting

Study finds zero-shot time-series foundation models underperform on CGM forecasting; fine-tuned Chronos-Bolt cuts RMSE up to 18.4% and dietary context adds signal.

The paper evaluates time-series foundation models for continuous glucose monitoring forecasting across eight public datasets covering Type 1 diabetes, Type 2 diabetes, and non-diabetes populations. Under a unified protocol, zero-shot foundation models did not consistently outperform baselines like Elastic Net and PatchTST, but lightweight fine-tuning did, with fine-tuned Chronos-Bolt reducing RMSE by 6.5%-18.4% in the T1D cohort and 8.6%-18.2% in the non-diabetes/T2D cohort. A residual-based fusion framework adding dietary context from CGMacros reduced overall RMSE by about 3% and postprandial RMSE by about 15% versus CGM-only baselines.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:43 UTC · 6d agoAI research

28

The OpenClaw 2.0 release moves your sessions into SQLite

OpenClaw 2.0 migrates AI agent sessions to SQLite, adds guided credential setup, and flags shared-session controls as not a security boundary.

OpenClaw 2.0, described as the largest update in the project's history, migrates sessions and transcripts into SQLite and adds guided setup that detects existing AI credentials from Codex, ChatGPT, Claude CLI sign-ins, API keys, and local Ollama or LM Studio models. The release expands multiplayer sharing while explicitly stating its permission controls are not tenant isolation or a security boundary, and that revoked access can briefly remain usable. Startup JavaScript requests fell from 140 to 45 and startup time from about 1.6 seconds to 575 milliseconds in simulated tests against a mocked Gateway. Automation wrappers must now inspect reported health because requesting --json does not waive risk acknowledgement.

Help Net Security·Aug 31, 00:00 UTC · 17d agoAI tools & infra1

35

Hackers Use Autonomous AI Agents to Harvest Thousands of Credentials in Under 6 Hours

Google Cloud documents a financially motivated actor using autonomous AI agents on a compromised cloud tenant to harvest 23,800+ credentials in under six hours.

Google Cloud reports that an attacker compromised a victim's cloud environment and deployed a multi-agent framework driven by preconfigured Markdown playbooks to autonomously handle vulnerability scanning, credential collection, error troubleshooting, and IP rotation. An exposed command-and-control server hosted the 'Recon' framework with a live dashboard managing over 23,800 harvested secrets, including cloud and AI-service API keys. The report also ties DUSTMAKER to UNC6780/TeamPCP, targeting AI development tools and CI/CD systems via trojanized MCP packages such as tiktoken_mcp. Google has disabled linked assets and updated protections after the actors' operational security failures.

Cyber Security News·Sep 15, 09:28 UTC · 1d agoThreat actor in the wild

72