ZeroHour

Search: “Nine”

24 stories

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

ProgramDistill is a benchmark evaluating coding agents on reconstructing web app features from reference applications, testing nine frontier agents.

ProgramDistill evaluates coding agents on features discovered through interaction with fully functional reference applications, factorizing apps into features with replayable behaviors verified via gold patches. Its mine-craft-patch pipeline discovered 1,975 replay-verified behaviors across 26 applications and built 4,063 tasks without human intervention. On cumulative full-application reconstruction workflows, GPT-6 Astra achieved 49.2% and Claude Opus 5 28.8% success. In partial reconstruction, success drops from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8.

Hugging Face daily papers · 1d agoAI research

Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents

XConf estimates LLM confidence from accumulated past episodes, improving calibration and discrimination across nine benchmarks at one-tenth self-consistency cost.

Researchers propose XConf, an experiential confidence estimator that augments current inference with a stored record of the model's graded past episodes, including reflections, stated confidence, outcomes, and lessons. A Recall stage retrieves episodes from similar tasks with similar stated confidence and reads off historical success rates, while a Reflect stage prompts the model to name recurring failure modes and restate confidence. Across nine benchmarks in reasoning, coding, multimodal QA, and interactive agents, and four models from three families, XConf beats or matches ten-sample self-consistency in AUROC on 23 of 24 comparisons with much lower ECE, at a tenth of the generation cost. For selective prediction, abstaining on the 10% least-confident episodes raises delivered success rate by up to 8.7 points on agent tasks.

Hugging Face daily papers · 2d agoAI research

Claude Fable 5.1 Solves the Cyphral Distich, a 370-year-old cipher

Claude Fable 5.1 solved Sir Thomas Urquhart's 370-year-old Cyphral Distich cipher, recovering a hidden royalist prayer for Charles II.

Vals AI reports that Claude Fable 5.1 solved the Cyphral Distich, a 64-number cryptogram from Sir Thomas Urquhart's Logopandecteision unsolved since 1653, in 44 minutes using 176k tokens with no human hints. The key insight was that the cipher's key was the book itself: each number indexes a word in the corresponding Proquiritation, taking the first letter, yielding 'O GOD UPHOLD KING CHARLS THE SECOND AND MAKE HIM THE SUPREME RULER OF THIS LAND'. The model also deciphered the larger Cyphral Octastich (285 numbers) from The Jewel (1652) using page-based word indexing, recovering all but nine letters of a royalist prayer. The puzzle had been listed among Klaus Schmeh's Top 50 unsolved encrypted messages.

Hacker News · AI · 3d agoAI researchHN 63↑ · 6 comments1· 1 read

Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

Climate-ModernBERT domain-adapted encoders reach 76.3 average F1 across nine climate benchmarks, 2.8 points above vanilla ModernBERT-Base.

The authors continue pretraining ModernBERT-Base on three climate corpora - academic text, climate-filtered web data, and synthetic documents - and compare joint mixtures against parameter-space merging of specialized checkpoints. The best model achieves 76.3 average F1 across nine climate NLP benchmarks, a 2.8-point improvement over the vanilla baseline. Academic climate corpora provide the strongest adaptation signal, and parameter-space merging outperforms joint multi-source training while preserving complementary corpus information; all variants are released.

arXiv cs.AI / cs.LG / cs.CL · 9d agoAI research

Building a Production Greek-English Speech Recognizer

Engineering report details Sophea, a production Greek-English ASR reaching 4.26% WER on public English sets via ROVER ensemble and data-pipeline calibration.

Across 23 training iterations, two architectures, and nine production gates, no single data composition passed all gates; a three-model ROVER ensemble reached 9 of 9 gates and cut overlapping-speech WER from 53.35% to 37.87%. Calibrating an audio-quality filter against in-domain anchors reduced discarded scored Greek audio from 98.7% to 10.6%, and a pre-registered ablation traced a hallucination defect to one training-data package. The sophea/asr-k1 preview arbiter lists 4.26% average WER on eight public English test sets and 25.88% WER on live Greek noisy traffic; no weights or training data are released.

Hugging Face daily papers · 6d agoAI research

An AI CAPTCHA solver talked itself out of the right answer

Bern researchers solved rotation CAPTCHAs in 0.006 seconds with classical computer vision, while Gemini 3.1 Pro needed 67 seconds and overruled correct tool answers.

Researchers at Bern University of Applied Sciences built a script using 1970s circle-detection math and signal matching that solved rotation CAPTCHAs in 0.006 seconds, scoring 10/10 on real-world puzzles. Frontier models fared poorly: Gemini 3.1 Pro scored 7/10 taking 67 seconds, while GPT-4o and Grok scored 1/10. When given the script's correct answer as a tool, Gemini overruled it and lost a fifth of its score; models could verbally describe targets, such as identifying a cyan ring, but could not produce accurate click coordinates. The paper also notes these no-JavaScript CAPTCHAs reduce tracking, leaving only shape-matching tasks classical vision solves easily.

Help Net Security · 15d agoAI research1

Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

Study shows radiology reporting-style variations in reference reports can flip rankings of chest X-ray report generation models; releases MIMIC-CXR-Ext-ReRef dataset.

The paper quantifies how variations in radiologists' reporting practices distort evaluation of radiology report generation (RRG) models, introducing a radiologist-informed taxonomy and the ReRef method for rewriting reference reports while preserving clinical meaning. On MIMIC-CXR with RadCliQ-v1, condensing normal-findings discussion caused Libra to drop from first to second while CheXOne rose from third to first among nine models. The authors release MIMIC-CXR-Ext-ReRef, a radiologist-validated dataset of 120 original/alternative reference pairs, arguing metrics conflate clinical correctness with stylistic conformity.

arXiv cs.AI / cs.LG / cs.CL · 19h agoAI research

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

Audit of 254 SWE-bench submissions finds top coding-agent entries statistically inseparable, so small leaderboard gaps no longer establish rank.

The paper audits 254 SWE-bench submissions across four splits without running models. On Verified, the top two entries each resolve 396 of 500 instances, and exact paired McNemar tests separate none of the 29 adjacent top-thirty pairs at alpha=0.05. Within-model scaffold score ranges reach 29.8 percentage points, versus an 8.8-point spread among the top thirty. The authors release a five-step audit protocol and recommend reporting comparison-set-specific resolution and model-scaffold provenance.

arXiv cs.AI / cs.LG / cs.CL · 1d agoAI research

LLM-Based Schema-Aware Split Learning for Privacy-Preserving Mental Distress Prediction Across Heterogeneous Surveys

Schema-aware split learning uses LLaMA-3.2-3B-Instruct as shared semantic encoder to harmonize heterogeneous mental-health surveys while raw data stays local.

The paper proposes a schema-aware split learning framework where an LLM serializes heterogeneous mental health survey records into natural language and is fine-tuned via LoRA, partitioned across client and server. Clients keep raw survey responses local and run only a lightweight front-end while the resource-intensive backbone runs server-side. Using LLaMA-3.2-3B-Instruct, the framework attains an average ANLS of 0.708 with 2,000 training samples, beats federated learning in eight of nine settings, and cuts per-client computation by three orders of magnitude while generalizing to unseen datasets.

arXiv cs.AI / cs.LG / cs.CL · 2d agoAI research

Deepmind's AlphaGenome Atlas maps every possible DNA change in the human genome

DeepMind's AlphaGenome Atlas precomputes impact predictions for ~9 billion human DNA variants in a 1-petabyte dataset; its AVI score beats CADD in benchmarks

Google DeepMind released the AlphaGenome Atlas, precomputing functional-effect predictions for roughly 9 billion human genome variants (about 27,000 prediction values per variant) in a one-petabyte dataset more than 30 times the size of the AlphaFold database. The accompanying AlphaGenome Variant Impact Score (AVI), a small neural network combining AlphaGenome, AlphaMissense and evolutionary conservation features (18 inputs versus CADD's 150+), outperformed existing tools on clinically classified variants, ranking causal variants in the top 50 candidates for 29.5% of solved GREGoR cases versus 12.5% for CADD. A GREGoR epilepsy case illustrates the impact: AVI elevated a previously unclear DNM1 splice variant that lab experiments confirmed as likely disease-causing. The atlas is available for noncommercial use via web portal, API and a Google Antigravity skill, with a commercial version planned through Google Cloud.

The Decoder · 7d agoAI research

Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

Audit of eight cybersecurity LLM benchmarks shows evaluation pipeline choices can swing scores by over 80 points and reshuffle most model rankings.

Researchers modeled eight cybersecurity benchmarks as configurable measurement pipelines and audited 10 proprietary, open-weight, and cybersecurity-specialized LLMs. They identified 15 systematic failure modes and showed a single pipeline choice can change a model's score by more than 80 percentage points and alter rankings; semantically similar task pairs rank the same models differently. Under a standardized harness, nine of 10 models shifted at least three ranks on at least one benchmark, motivating pipeline-aware auditing for reliable model evaluation.

arXiv cs.CR · 8d agoAI research1

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

DataFlex-RL benchmark of 13 RLVR data policies on Qwen2.5-7B finds none reproducibly beats uniform sampling under matched GRPO training.

DataFlex-RL is an evaluation platform comparing rollout-selection, reweighting, and mixture data policies for RLVR under a common GRPO recipe. Across 13 configurations and 12 matched seeds with Qwen2.5-7B-Base on 12 math, logic, and science benchmarks, uniform GRPO improved domain-balanced accuracy by 7.76 points, but no alternative policy achieved a statistically significant improvement. A corrected 12-seed Llama-3.1-8B-Base extension found no consistent winner, and math-heavy evaluation summaries were negatively correlated (-0.33) with domain-balanced summaries.

Hugging Face daily papers · 12d agoAI research

What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation

A new benchmark shows LLMs reach 68.3-93% accuracy propagating local revisions across conversationally generated artifacts, with parallel-sample selection most cost-effective.

The paper introduces a benchmark for revision propagation: when users request a local change, LLMs must identify dependencies and update all affected parts of an artifact generated through conversation, where context lives in the chat history. Nine revision methods, including sequential reflection and parallel sampling variants, were evaluated on gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b. Baselines scored 68.3-93% accuracy, and selecting among three parallel samples via LLM-based or medoid selection improved accuracy by 2.2-9.7% as the most cost-effective test-time compute strategy. Code and dataset are released.

Hugging Face daily papers · 14d agoAI research