ZeroHour

Search: “annotation”

218 stories

AI labs have a data trust problem that their policies haven't solved

Nvidia, Palantir, and Booz Allen restrict Anthropic's Fable over data-retention distrust, exposing gaps in AI labs' customer data policies.

Nvidia limits Anthropic's Fable to non-sensitive work and runs its own Nemotron models for internal tasks, while Palantir blocks Fable deployment until Anthropic grants irrevocable zero-data-retention guarantees, and Booz Allen bans it for proprietary cybersecurity work. John Schulman and researcher Sarah Hooker explain that labs can still extract customer IP from metadata, user traces, and synthetic data even under zero data retention. The trust crisis crystallized around Tristan Buckmaster's accusation that OpenAI's Codex absorbed his Navier-Stokes drafts, though OpenAI later stated his prompts could not have influenced its model.

The Decoder · 20h agoAI industry

Verifiable Social Reasoning for LLM Assistants

Fuse, a multi-agent simulation with hidden motives, evaluates LLM social reasoning, revealing compounding difficulty from user mediation and bias sensitivity.

Fuse is a multi-agent simulation framework in which a target agent with a hidden motive interacts with other agents including one representing the user, who consults the evaluated assistant to infer the motive, providing verifiable ground truth by construction. Simulation faithfulness is validated through a human study with 24k annotations. Applied to 12 LLMs, it shows user mediation compounds social reasoning difficulty, models are systematically sensitive to biased user framing, models may need more details than humans, and longer conversations do not always improve performance. The framework and a 21k-example dataset are open-sourced.

arXiv cs.AI / cs.LG / cs.CL · 20h agoAI research

Decomposition Buys Integrity, Not Yield

Study of 600 production deep-research traces finds agent-tree decomposition loses findings at rate N^(1-δ); flat architectures maximize yield.

The paper models multi-agent decomposition as a tree where an agent holding b items retains each with probability r(b); with r(b)=1/b every tree delivers exactly one finding regardless of shape. Analysis of 600 production deep-research traces estimates delta=0.34 retention decay, and 1,012 annotated traces show one brief in sixteen goes off-target per tier, giving an alignment penalty of 0.536. Depth still cuts root context exposure from N to N^(1/k) and is cheaper at scale, with a hazard model over 743,819 production tool calls showing delegation is an opening move rather than a response to filling context.

arXiv cs.AI / cs.LG / cs.CL · 21h agoAI research

Evaluating the NIST Bugs Framework Against CWE as a Successor for Automated Vulnerability Classification

NIST Bugs Framework evaluation shows it is more structured and automation-friendly than CWE for automated vulnerability classification, with gaps in attribute guidance.

The paper evaluates NIST SP 800-231's Bugs Framework (BF) against CWE as a target for automated CVE classification using a systematically screened corpus of CVE-to-CWE research. An inter-rater study with 2 subject-matter experts mapping 13 CVEs showed strong agreement on BF's cause and operation axes but only fair agreement on the attribute axis. Automated classification was tested across two LLM deployments under different budgets, and findings support BF as more structured and automation-friendly than CWE, though gaps include under-specified attribute guidance and missing fix commits for closed-source software.

arXiv cs.CR · 1d agoResearch1

Agent Harness vs Agent Framework vs MCP: Which Layer Owns the Loop, State, Tools, Permissions, and Recovery

Architecture explainer separates agent harnesses, frameworks, and MCP by which layer owns the loop, state, permissions, and recovery.

The article distinguishes agent harnesses (OpenAI Codex, Claude Agent SDK), which own the execution loop, sandbox, permission model, and recovery; frameworks (LangGraph, OpenAI Agents SDK, Microsoft Agent Framework), which supply composable primitives; and MCP, a stateless JSON-RPC wire protocol governed by the Linux Foundation's Agentic AI Foundation since December 2025. An ownership matrix maps the execution loop, state, tool transport, permissions, recovery, sandboxing, and multi-agent orchestration to each layer. The 2026-07-28 MCP specification made the protocol fully stateless, retiring the initialize handshake and session headers.

MarkTechPost · 1d agoAI research1

Inside ‘Project Lily’: The Humans Reading Your ChatGPT Chats

404 Media reveals OpenAI's 'Project Lily' has hundreds of contractors reading real ChatGPT user prompts, exposing sensitive personal data despite privacy filters.

404 Media reports that OpenAI employs hundreds of contractors who read real ChatGPT user prompts, including whole conversations, to rate and critique the chatbot's responses across a user base of over 900 million. Prompts are anonymized and run through OpenAI's Privacy Filter model, but the company acknowledged sensitive personal details can still reach reviewers, and 'user memories summaries' may reveal a user's location and personal context. The review work includes training ChatGPT to be less sycophantic and to stop anthropomorphizing itself, following lawsuits linking the sycophantic 4o model to multiple suicides. Anthropic confirmed it also uses human review to improve its models, and OpenAI's 'improve the model for everyone' data-sharing setting is on by default for free, Plus, and Pro users.

404 Media · 1d agoAI safety & security

Approval Integrity and Recovery in LLM Answer Publication

Study measures approval integrity in Lightcap LLM answer publication, finding the 14B response-act checker accepts 291 of 302 unsupported answers.

The study evaluates exact-content binding, authorization freshness, and checkpoint recovery in Lightcap's publication enforcement using 3,600 assessments over 900 human-annotated RAGTruth responses from three Ministral models. The production 14B response-act checker accepts 291 of 302 unsupported answers versus 41 for a direct-grounding baseline, with supported-answer retention of 95.2% versus 66.9%. A stateful recheck-recovery policy increases exact-match error by 9.23 percentage points relative to initial checkpoints, and controlled evidence-fingerprint changes expose asymmetric freshness enforcement between publication and recovery. A separate BIPIA prompt-injection experiment records zero target insertions among 266 valid editor outputs.

arXiv cs.CR · 2d agoAI safety & security

How Fyxer built an AI executive assistant people trust

Fyxer details its OpenAI-powered AI executive assistant, orchestrating 30-50 specialized models trained on 500,000+ hours of assistant workflows.

OpenAI published a case study on Fyxer, whose AI executive assistant orchestrates 30-50 specialized OpenAI models trained on more than 500,000 hours of annotated executive assistant workflows. The system uses supervised fine-tuning, LoRA, and Direct Preference Optimization on user edits, and 53% of AI-generated email drafts are accepted as written. Fyxer's annual recurring revenue grew from $1 million to $32 million during 2025.

OpenAI News · 2d agoAI industry

HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness

Researchers introduce HarnessVLN, a zero-shot training-free agent harness that sets new training-free SOTA on vision-language navigation benchmarks including R2R and HM3D.

HarnessVLN is a zero-shot, training-free framework for embodied vision-language navigation that coordinates perception, retrieval, grounding, navigation, recovery, and termination through a unified tool interface. It validates planner proposals against spatial evidence, geometric feasibility, and subgoal consistency, using hierarchical event memory and a persistent Spatiotemporal Graph that stores reusable spatial evidence and failure annotations. It reports success rates of 60.8% on R2R, 53.9% on RxR, 76.0% on HM3D-v2, and 59.3% on HM3D-OVON, surpassing prior training-free state of the art, with real-world humanoid deployment demonstrated.

Hugging Face daily papers · 2d agoAI research

LynnReal-Omni: Native multi-modal Video Generation for Agentic Visual Workflows

LynnReal-Omni unifies controllable video generation tasks in a 32B multimodal diffusion transformer, with a 27B Flash variant rendering 540p clips in 377 ms.

LynnReal-Omni is a native multimodal video generation framework built on a 32B shared multimodal diffusion transformer unifying text-to-video, image-conditioned generation, reference guidance, structural control, editing, restoration and long-video generation, accepting heterogeneous inputs like 3D renders and game recordings for agentic visual workflows. A dedicated 27B Flash model enables real-time rendering, producing a 22-frame 540p video in 377 ms on one H100 versus 843 ms for the full model. The work introduces a curated multi-shot audiovisual data pipeline and MSAVP, a 100-prompt, 20-metric evaluation design covering instruction following, plausibility, visual quality, temporal behavior and audio coordination.

Hugging Face daily papers · 2d agoAI research

AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video

AlayaVista is a camera-controllable streaming video world model that decouples panoramic scene evolution from perspective synthesis, trained on a 1,318-hour 4K dataset.

AlayaVista builds a 360-degree scene prior from a single perspective image, evolves it as a camera-conditioned panoramic latent state, and maps it to perspective video via a latent viewport renderer plus a perspective refiner. Chunk-autoregressive generation and few-step distillation enable efficient streaming. The authors introduce MUGEN, a real-world panoramic video dataset with 1,318 hours of at-least-4K video and rich semantic and geometric annotations.

Hugging Face daily papers · 3d agoAI research

Google Research Releases ToolGrad: Answer-First Framework Hits 99.8% Pass Rate for Tool-Use Data Generation

Google Research and partners introduce ToolGrad, a verified tool-chain-first data generation framework reaching 99.8% pass rate and boosting Gemma-3-12B to 83.1 on BFCL.

Researchers from Google, the University of Tokyo, RIKEN AIP, and Tohoku University released ToolGrad, which inverts query-first tool-use data generation by executing and verifying API chains before annotating them with user queries. On the ToolBench database of 16,000+ APIs, ToolGrad raised generation pass rate from 63.8% to 99.8% while increasing tool uses per sample from 2.1 to 3.4 and cutting tool-use steps from 34.3 to 20.0. Fine-tuning Gemma-3 at 1B, 4B, and 12B parameters on the 500-sample ToolGrad-500 dataset lifted ToolGrad-12B to 83.1 on the Berkeley Function Calling Leaderboard, near Gemini 2.5 Pro at 83.2 and ahead of GPT-5 at 74.4. Code is Apache-2.0, with the dataset, PyPI package, and models available on Hugging Face.

MarkTechPost · 5d agoAI research1

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

Continual Search framework iteratively prompts LLM judges to keep searching agent execution logs, boosting long-horizon failure root-cause attribution accuracy.

The paper frames automated root-cause attribution (RCA) for long-horizon AI agent failures as a search problem, since relevant evidence is sparse and distributed across massive execution traces. The authors propose Continual Search, an iterative framework that nudges an LLM judge across successive turns to keep hunting unresolved diagnostic evidence instead of settling on an early plausible diagnosis. They introduce MegaRCA-Mix, a benchmark of 50 human-annotated failure trials on long-horizon, execution-heavy tasks. On MegaRCA-Mix, Continual Search improves GPT-5.5's F1 from 0.349 to 0.498 (over 40% gain), and lower-tier models can surpass higher-tier counterparts when search is effective.

Hugging Face daily papers · 5d agoAI research1

TART: A Modular Tool for Technique-Aware Audio-to-Tablature Guitar Transcription

TART pipeline transcribes guitar audio to tablature with fingering and technique labels, outperforming prior baselines by up to 8.5 Tab F1 points.

TART is a modular four-stage pipeline for audio-to-tablature guitar transcription, combining an audio-to-MIDI transcription model, an expressive technique classifier, an audio-conditioned T5 encoder-decoder for string-fret assignment, and an automated tablature generator. In zero-shot evaluation on GuitarSet, EGDB, and two noisy augmented benchmarks, it averages 81.35% audio-to-MIDI F50 (+6.67 over the best baseline), 71.8% string-fret Tab F1 (+8.5), and 54.08% end-to-end Tab F1. The authors state it is the first framework to generate tablature with both fingering and expressive technique annotations directly from guitar audio.

arXiv cs.AI / cs.LG / cs.CL · 5d agoAI research2

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

Researchers release IndicTriMix benchmarks and fine-tuned MuRIL and XLM-RoBERTa models for token-level language identification in tri-language code-mixed text.

The paper formulates token-level language identification in code-mixed text as a sequence labeling task and fine-tunes MuRIL and XLM-RoBERTa transformer models for Indian languages. It evaluates on Hindi, Gujarati, and Bengali configurations with manually annotated test sets and proposes two code-mixed generation approaches using parallel trilingual sentences. A public benchmark, annotated test sets, and fine-tuned models are released for reproducibility.

arXiv cs.AI / cs.LG / cs.CL · 5d agoAI research1

Skild AI Taps NVIDIA Physical AI to Teach Robots New Tasks From a Single Video

Skild AI launched its S1 robot foundation model, built on NVIDIA infrastructure, that learns long-horizon industrial tasks from a single video.

Skild AI's S1 model uses in-context learning from one video demonstration to execute unfamiliar multistep tasks lasting up to 10 minutes without weight updates or task-specific post-training. In tests on new tasks it achieved about 66% per-step success versus 9% for a comparable AI system, and one video demonstration was estimated to match roughly 380 hands-on training examples. The company reached a $100 million annual revenue run rate with more than 60 deployment partnerships, and with NVIDIA and Foxconn deploys the Skild Brain on dual-arm manipulators assembling NVIDIA Blackwell systems. Training and validation rely on NVIDIA Isaac Lab, Isaac Sim, Omniverse, Cosmos and the Newton physics engine.

NVIDIA Blog · 5d agoModel release

Show HN: MultiMatte, a Promptable Image Background Removal Model

Feyn releases MultiMatte, a promptable background-removal model fine-tuned from Meta's SAM 3 via LoRA, outputting alpha mattes that beat SAM 3 on segmentation benchmarks.

Feyn introduced MultiMatte, a promptable image background-removal model built on Meta's SAM 3 (860M parameters). It modifies only 19.49M parameters (2.27%) using a rank-16 LoRA adapter and replaces binary masks with alpha mattes to handle fuzzy boundaries like hair. On the DIS-VD benchmark it scores 0.901 S-measure versus SAM 3's 0.667, and it improves on SAM 3 across all twelve evaluated splits. Training used 19,953 images for 14,000 steps with focal and Dice loss, and the merged weights are available via the nobg library and a web demo.

Introducing ChatGPT for Financial Services

OpenAI launches ChatGPT for Financial Services, pairing built-in market data with GPT-6 Astra for banking research workflows.

OpenAI introduced ChatGPT for Financial Services, a tailored ChatGPT Work experience shaped by design partners Morgan Stanley and Evercore, targeting investment banking and equity research. It bundles premium data from Daloopa, PitchBook, LSEG News, and Crunchbase hosted on OpenAI infrastructure with granular citations, optimized MCP connectors for S&P Global and FactSet, and 50+ connectors, plus planned entitlement integrations with S&P Capital IQ, LSEG, MSCI, Dow Jones Factiva, and Moody's. It runs GPT-6 Astra, which OpenAI claims is state of the art in information retrieval, financial reasoning, and artifact generation, and includes enterprise controls such as SAML SSO, SCIM, role-based access, and no default training on firm data.

OpenAI Newsupdated · 5d agofirst · 6d agoAI industry 3 sources

Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision

ECCV 2026 challenge winner reformulates egocentric intervention timing as single-token classification, boosting macro-F1 by 0.249 over free-form generation.

The paper describes the winning submission to the EgoProactive track of the ECCV 2026 Wearable AI Challenge, ranking first in the large-model division and second in the <=2B division. The method reformulates intervention timing as single-token yes/no classification, improving macro-F1 by 0.249 and G-mean by 0.30 over free-form generation. Supervision generated by a tool-calling video agent transferred better than a narration-only dataset that was four times larger and ten times cheaper, suggesting visual grounding matters more than annotation volume.

Hugging Face daily papers · 6d agoAI research

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

IdeaAMBIG benchmark with 660 instances measures whether LLMs can spot and fix underspecified research-method details for faithful implementation.

Researchers introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances (163 real-world gaps from reproducibility reports and GitHub issues, 497 controlled synthetic gaps) built from papers, codebases, and reproduction artifacts. It evaluates codification-readiness assessment, defect localization, and clarification action generation. Across 13 LLMs, the best model achieved only 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% clarification success when given the annotated defect. An oracle study showed gold resolutions raise the codification-ready rate from 14% to 98%, identifying defect localization as the main bottleneck.

arXiv cs.AI / cs.LG / cs.CL · 6d agoAI research2

Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation

Referee-Based Quality Estimation flags unreliable polyp segmentations at inference without ground truth, reaching ROC-AUC 0.960 with SegFormer-B0 referees.

RBQE measures agreement between a primary segmentation model and an independently trained referee on a 1,223-image external benchmark drawn from four public datasets. A cross-architecture SegFormer-B0 referee achieves the strongest signal (ROC-AUC 0.960), beating a Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol. Excluding trivially separable empty-mask cases, ROC-AUC falls to 0.876 (SegFormer-B0) and 0.783 (same-architecture control), but RBQE's margin over baselines widens. Progressive rejection of low-agreement predictions increases mean Dice of retained outputs, supporting selective prediction at the cost of one extra forward pass.

arXiv cs.AI / cs.LG / cs.CL · 6d agoAI research

Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalization

ModerationBench shows foundation models can nearly triple Bluesky's moderation F1 (0.60 vs 0.22), with instruction- and example-driven guidance performing comparably.

Researchers built ModerationBench, a new benchmark of 4,000 manually annotated in-the-wild posts from Bluesky, to test whether foundation models can reliably operationalize content moderation policies. They systematically compare instruction-driven guidance (reasoning from policy precepts) with example-driven guidance (generalizing from precedents) for Vision-Language Models. Both paradigms achieve comparable peak effectiveness, and foundation models nearly triple the F1 of Bluesky's deployed moderation system on Random Posts (0.60 vs 0.22).

arXiv cs.AI / cs.LG / cs.CL · 6d agoAI research1

NVIDIA Brings Real-Time AI to Broadcast, Sports and Global Streaming at IBC

NVIDIA expanded its AI for Media suite at IBC 2026, adding NIM microservices for synthetic video detection, body pose, frame generation, upscaling and HDR.

At IBC 2026 in Amsterdam, NVIDIA announced a major expansion of NVIDIA AI for Media, a collection of GPU-accelerated SDKs, NIM microservices and blueprints for broadcast and streaming workflows. The Synthetic Video Detector (SVD) NIM microservice reaches 99.3% accuracy on text-to-video and 97.7% on image-to-video content, while Video Frame Generation boosts frame rates 2x-4x and Video Super Resolution adds 10-bit support; TrueHDR converts SDR to HDR at up to roughly 2,000 nits. Partners including Dalet, TwelveLabs, Wowza, Vizrt and Ross Video are integrating the new services into verification, compliance and live-production workflows.

NVIDIA Blog · 6d agoAI industry

ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

ReactHuman benchmark tests whether multimodal LLMs react safely to sudden household hazards; seven evaluated models mishandle roughly one hazard in three.

ReactHuman is the first physics-grounded benchmark for human-like reactive decision-making, placing a multimodal LLM as the brain of a simulated humanoid facing 17 event families of sudden household hazards across over 1,000 bit-for-bit reproducible scenes with annotation-free ground truth from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics. A five-metric suite scores each reaction along reasonable, safe, and physically grounded axes, and every committed plan is physically executed. Seven representative MLLMs mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale; none of these failures shrink with model scale.

Hugging Face daily papers · 7d agoAI research