Search: “Wagemole”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

North Korean Job Fraud Expands Beyond IT Into Healthcare and Sales

North Korea's IT worker scheme (Famous Chollima/PurpleDelta) has expanded from IT into healthcare, sales, and financial services roles worldwide.

Huntress and Recorded Future documented DPRK-linked fraudulent workers landing remote jobs beyond IT, including at an Australian healthcare company, a financial services firm, and a sales hire with a stolen identity. The scheme, tracked as Famous Chollima, Jasper Sleet, Nickel Tapestry, PurpleDelta, UNC5267, and Wagemole, uses forged identity documents, VPNs, proxies, and laptop farms with PiKVM and capture cards to fund Pyongyang's weapons programs. Recorded Future found the PurpleDelta cluster applied to 1,100+ companies between late 2024 and early 2025 with 22 fabricated personas, some AI-generated, using ChatGPT and AI transcription during interviews. Analysts assess the activity is ongoing and likely to expand in scale and sophistication.

The Hacker News·Sep 1, 06:34 UTC · 15d agoThreat actor in the wild

65

When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control

Chain-of-Self-Questioning prompting cuts LLM wrong-answer commitments 32% relative while raising answered accuracy, holding across eleven model families.

The paper introduces Chain-of-Self-Questioning (CoSQ), a prompt-only framework that makes LLM answer commitment conditional on an explicit assessment of the information required to answer. On an 817-item TruthfulQA multiple-choice set, Grounded-CoSQ at τ=0.90 reduced mean unconditional wrong-commitment rate from 13.1% under chain-of-thought to 8.9% (a 32.1% relative reduction), while raising answered accuracy from 86.9% to 89.7% at 87.6% coverage. Improvements held across eleven open-weight and hosted model families and at every evaluated threshold, with convergent evidence from a Natural Questions short-answer evaluation.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:52 UTC · 1d agoAI research

30

Engineered Persuasion: Evaluating Personalized Pretexts in LLM-Generated Spear Phishing

A study of 180 US workers found each LLM phishing personalization level raised click-intention odds by 28%, but credibility depends on context fit.

The arXiv paper evaluates how personalized pretexts in LLM-generated spear phishing affect perceived credibility, using 180 US working adults across 1,436 evaluations of emails with four cumulative personalization levels, from workplace context to shared-project details. Convincingness rose 2.40 points per level in sensitivity analysis and click-intention odds increased 28% per level, while non-clickers shifted toward deleting rather than reporting. Qualitative coding showed details matching the recipient's role and routines supported credibility, whereas incorrect, vague, or channel-inappropriate details raised suspicion. The authors argue personalization effectiveness depends on pretext fit, with implications for workplace security training.

arXiv cs.CR·Sep 3, 19:21 UTC · 13d agoResearch

35

Guiding Worker Self-Selection in Crowdsourcing Contests: An LLM-Augmented Algorithmic Approach

Researchers introduce GRAF, a greedy framework for crowdsourcing contest self-selection, and LLMScore, an LLM-driven method that auto-designs its scoring algorithm.

The paper studies self-selection in Tullock contests (SSTC), where workers choose contests and then compete within them. GRAF is a greedy polynomial-time framework that orders workers by a score vector with zero worker regret and platform optimality guarantees in special cases. LLMScore is an LLM-driven evolutionary framework that produces human-readable, inspectable scoring code, jointly optimizing platform utility and worker satisfaction. Across 1,000 synthetic instances in four settings, GRAF with LLMScore achieves high-quality, often near-optimal outcomes with low worker regret, transferring from small training instances to larger, structurally different settings.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 16:51 UTC · 9d agoAI research1

22

Bad Likert Judge: A Novel Multi-Turn Technique to Jailbreak LLMs by Misusing Their Evaluation Capability

Unit 42 details the Bad Likert Judge multi-turn jailbreak that abuses LLMs' evaluation capability, raising attack success rates over 60% across six frontier models.

Palo Alto Networks Unit 42 describes the Bad Likert Judge technique, a multi-turn jailbreak that asks a target LLM to act as a Likert-scale judge scoring the harmfulness of example responses. The highest-rated example in each scale can carry harmful content, bypassing the model's internal guardrails. Testing across six state-of-the-art text-generation LLMs showed an average attack success rate increase of more than 60% versus plain attack prompts, with tested models anonymized. The technique targets edge cases rather than typical use, and the article positions the work as guidance for defenders on potential jailbreak risks.

Palo Alto Unit 42·Aug 17, 12:34 UTC · Aug 17, 2026AI safety & security

60

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled autoregressive testbed shows validation losses must be analyzed per task, and image tokenizer choice affects joint multimodal text modeling.

Researchers built a pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models, tracking task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They found that losses exhibit distinct scaling behavior per task and rank tokenizers differently, and that I2T loss over a shared text vocabulary gives a more consistent loss–performance signal than T2I loss. Better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and tokenizer choice can affect text modeling under joint optimization. Case studies examine the discriminator, semantic supervision, and vocabulary size design axes.

Hugging Face daily papers·Sep 7, 20:00 UTC · 9d agoAI research1

40

Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

Evaluation of twelve LLMs on 222 clinical questions shows verbatim quotes rarely substantiate claims; claude-opus-5 fully substantiates only 37.1%.

The authors build a standardized harness over four clinical practice guidelines and evaluate twelve LLMs on 222 synthetic clinical questions, measuring citation attachment, verbatim quote production, and claim substantiation. Most models attach verbatim quotes to over 90% of claims from prompting alone, though lightweight models like claude-haiku-4.5 struggle. Quotes frequently fail to substantiate claims: claude-opus-5 quotes 98.0% of claims but fully substantiates only 37.1%, exposing a capability gap for verifiable clinical QA.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:53 UTC · 2d agoAI research

38

LLMs and Contextual Integrity

Bruce Schneier highlights two papers: the CIMemories benchmark shows frontier LLMs leak memory attributes up to 69%, and an RL method reduces inappropriate disclosures.

Bruce Schneier discusses contextual integrity in LLMs, referencing the CIMemories benchmark, which uses synthetic profiles with 100+ attributes per user to test whether models with persistent memory disclose sensitive information appropriately. Evaluation showed frontier models exhibit up to 69% attribute-level violations, with GPT-5's violation rate rising from 0.1% to 9.6% across 40 tasks and reaching 25.1% with repeated prompting, showing unstable leakage behavior. A second paper introduces a reinforcement learning framework trained on a synthetic 700-example dataset that substantially reduces inappropriate disclosure while maintaining task performance, with improvements transferring to the human-annotated PrivacyLens benchmark.

Schneier on Security·Aug 18, 10:40 UTC · 29d agoAI safety & security1

48

ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

ReactHuman benchmark tests whether multimodal LLMs react safely to sudden household hazards; seven evaluated models mishandle roughly one hazard in three.

ReactHuman is the first physics-grounded benchmark for human-like reactive decision-making, placing a multimodal LLM as the brain of a simulated humanoid facing 17 event families of sudden household hazards across over 1,000 bit-for-bit reproducible scenes with annotation-free ground truth from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics. A five-metric suite scores each reaction along reasonable, safe, and physically grounded axes, and every committed plan is physically executed. Seven representative MLLMs mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale; none of these failures shrink with model scale.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research

42

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

MetroLLM-Bench is a 955-case benchmark testing language models as transit kiosk tool-calling runtimes across six real metro systems.

The benchmark covers 37-414-station metro systems and eleven task categories including routing, fare calculation, disruptions, accessibility, and adversarial input, with 14 deterministic and 8 semantic scoring components. Of 26 models from six vendors, a PEFT-tuned 4B Qwen 3.5 student scored 91.3 on Tier 1, exceeding GPT-5.6 (90.6/90.0), while Muse Glimmer 30B led the composite ranking. A deterministic rule-based baseline reached 84.6, and PEFT gains over base models shrank from +7.03 points at 2B to -0.91 at 27B.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research1

30

Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM

French BabyLM entry METRON-FR (125M GPT-2, 92.47M words) shows tokenizer artifacts dominate child-scale zero-shot evaluation; proposes standard diagnostics.

METRON-FR is a 125M-parameter GPT-2 pretrained on 92.47M French words, submitted to the BabyLM 2026 Strict track, scoring 85.97% on the native Quebec-French QFrBLiMP benchmark and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE protocol combining French task-data translation with rank-16 LoRA shows relational tasks gain while world-knowledge tasks regress. Bilingual Lexicon Induction reaches p@1 of 68.84%, 18x above chance, and ablations show single-token zero-shot scoring is dominated by tokenizer and template artifacts at child scale.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 16:43 UTC · 1d agoAI research

12

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

IdeaAMBIG benchmark with 660 instances measures whether LLMs can spot and fix underspecified research-method details for faithful implementation.

Researchers introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances (163 real-world gaps from reproducibility reports and GitHub issues, 497 controlled synthetic gaps) built from papers, codebases, and reproduction artifacts. It evaluates codification-readiness assessment, defect localization, and clarification action generation. Across 13 LLMs, the best model achieved only 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% clarification success when given the annotated defect. An oracle study showed gold resolutions raise the codification-ready rate from 14% to 98%, identifying defect localization as the main bottleneck.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 17:59 UTC · 7d agoAI research2

28

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.

Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:57 UTC · 8d agoAI research1

30

Before You Poll with LLMs: A Deliberative Diagnostic Framework

Deliberative diagnostic shows all five tested frontier LLMs misrepresent human belief shifts after arguments, with GPT-5.1 reversing on outgroup questions.

The Deliberative Polling Diagnostic Framework compares human and LLM persona belief shifts after identical informational interventions, using data from America in One Room (526 personas, 72 questions). All five frontier models tested failed uniquely: GPT-5.1 exhibited partisan reversal (80% on outgroup vs 26% on policy questions), Gemini 2.0 Flash, Claude Sonnet 4.5 and Llama 3.3 70B overshot at 5-7x human magnitude, and DeepSeek V3 showed near-zero change (rigidity). The authors term the underlying signature 'self-sycophancy', conformity to the model's internal persona stereotype rather than reasoning from provided information.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 16:41 UTC · 2d agoAI research

40

PentestGPT: Open-source automated penetration testing agentic framework

Open-source PentestGPT runs autonomous LLM-driven penetration tests via Claude Code and Codex, with legacy human-in-the-loop mode supporting many providers.

PentestGPT, originally published at USENIX Security 2024 by Gelei Deng and colleagues, is an open-source framework that lets a large language model autonomously run penetration testing stages (recon, exploit, walkthrough) with no human in the loop, driving Claude Code or Codex CLIs. A legacy interactive mode uses three cooperating LLM sessions maintaining a Pentesting Task Tree and supports OpenAI, Anthropic, Google Gemini, DeepSeek, xAI, Qwen, Moonshot, and local models via Ollama. The tool sends anonymous telemetry to Langfuse by default, excluding command outputs, credentials, and flags, and is available free on GitHub.

Help Net Security·Aug 12, 00:00 UTC · Aug 12, 2026Tools1

48

PuzzleMask: Abusing Plain Prose as a Covert AI Attack Vector

Check Point details PuzzleMask, a plain-prose technique that bypasses LLM gatekeeper policy checks, letting hidden payloads reach target models unreviewed.

Check Point Research describes PuzzleMask, a prompt-crafting technique that hides policy-violating payloads inside plain-English prose wrappers, bypassing quick LLM-based policy checks without emojis, Base64, or invisible formatting. The researchers tested 23 automated prompts against gatekeepers including GPT-4o-mini, GPT-OSS-Safeguard 20b, Claude 3 Haiku, and Llama Guard 3, and all were classified as safe despite policies that flagged the plain versions. When submitted to GPT-5 in thinking-high mode with a Python interpreter, the target model extracted and acted on the payload in over 90% of trials. The technique is not itself a jailbreak but can carry a jailbreak prompt as payload; mitigations include input paraphrasing, hardened gatekeeper policies, and output monitoring.

Check Point Research·updated Sep 11, 05:53 UTC · 5d ago·first Sep 10, 14:32 UTC · 6d agoAI safety & security 2 sources

55

Nebulon Enterprise Simulated Threats for Phishing Research (NEST-Phish): A Synthetic Enterprise Phishing Email Dataset for Behavioral and Machine-Learning Research

Researchers release NEST-Phish, a synthetic enterprise phishing email dataset with matched legitimate and phishing emails and cue annotations for detection research.

Academic researchers introduce NEST-Phish, a publicly released synthetic enterprise phishing email dataset built around a fictitious organization named Nebulon. It contains matched synthetic legitimate and phishing emails across a broad set of workplace communication themes, each with interpretable phishing-cue annotations. Human-subject categorizations and supervised classifier evaluations indicate the dataset supports meaningful variation in phishing judgments and provides learnable signal for detection models. It is intended to support work on phishing detection, human susceptibility, explainability, and benchmark development.

arXiv cs.CR·Sep 3, 20:58 UTC · 13d agoResearch

35

From Reweighting to Rewriting: Unlocking the Intervention Effects of Influential Samples in Training Data Attribution

Study shows rewriting responses of influence-selected training examples shifts LLM behavior more strongly than reweighting the same samples.

The paper examines training data attribution, arguing that influence functions identify high-leverage examples whose value goes unrealized under conventional weight-based reweighting interventions. It introduces influence-guided response rewriting, which replaces the responses of influence-selected examples with behavior-aligned or behavior-opposed supervision while keeping instructions fixed, tested across four open-weight LLMs using epistemic abstention as the primary testbed. Rewriting produces stronger, more persistent, and bidirectional behavioral shifts, including on safety refusal, while reweighting the same examples yields weak, inconsistent effects. The results motivate intervention-aware evaluation of TDA methods.

Hugging Face daily papers·Sep 1, 20:00 UTC · 15d agoAI research

30

Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training

Exploration-guided prompt scaffolding rewrites training prompts by Exploration Potential Score, boosting multimodal RL post-training accuracy up to 11.5%.

The paper proposes dynamically adapting the training prompt distribution during online RL post-training of multimodal LLMs using the Exploration Potential Score (EPS), a lightweight rollout-based proxy for prompt utility computed from on-policy statistics with no additional overhead. Rather than discarding low-utility prompts, a teacher model generates scaffolded rewrites that preserve task intent while making training more informative. Integrated with GRPO on Geo3K and MMK12, the method achieves up to 9.7% relative in-domain improvement plus 11.5% on MathVision and 11.1% on MMMU-Pro.

Hugging Face daily papers·Sep 13, 20:00 UTC · 3d agoAI research

35

Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models

New ROBORMBENCH benchmark shows vision-language reward models can flip robot success/failure judgments when goal instructions are paraphrased.

The authors show that paraphrasing the instruction alone can substantially change progress scores from VLM reward models, even flipping identical robot trajectories between failure and success. ROBORMBENCH comprises 2,390 real-robot trajectories with ground-truth progress labels and 21,673 verified paraphrases covering lexical, syntactic, and action-goal rewrites. Instability is widespread across proprietary and open-source VLMs, grows with more divergent rewrites, and is not reliably reduced by scale or explicit reasoning, while trajectory-grounded dedicated reward models are markedly more stable.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:47 UTC · 12d agoAI research1

35

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

IdeaAMBIG benchmark of 660 specification-gap instances shows LLMs localize implementation-critical research gaps poorly, with best model at 9.6% defect recovery.

IdeaAMBIG is a benchmark of 660 evidence-grounded instances evaluating whether research-method specifications provide enough information for faithful implementation: 163 real-world gaps from reproducibility reports and GitHub issues plus 497 controlled synthetic gaps. It tests codification-readiness assessment, defect localization, and clarification action generation across 13 LLMs. The best model achieves only a 9.6% Macro Defect Recovery Rate on real-world instances, though 80.6% clarification success when given the annotated defect, and an oracle study shows gold resolutions raise codification-ready rates from 14% to 98%. Defect localization emerges as the main bottleneck across all evaluated models.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research

30

Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence

New framework tests whether LLM-cited explanation factors are necessary or sufficient, finding weak correlation across Claude, GPT, and Gemini models.

An arXiv paper introduces black-box intervention tests measuring whether factors LLMs cite in their explanations are necessary or sufficient for their outputs in agent oversight workflows. Across eight models from the Claude, GPT, and Gemini families, Spearman correlations between cited rankings and measured influence ranged from 0.349-0.354 (advisor recommendation) to 0.431-0.580 (prompt monitoring). Uncited factors scored above the lowest cited factor in up to 57.6% of advisor responses, showing cited top-three factors do not reliably identify the most influential inputs.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:37 UTC · 12d agoAI safety & security1

44

BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure

BenchShield uses lifecycle-model-backed instrumentation to detect reward hacking in LLM-agent benchmarks, lifting full-chain recall to 77-100% at up to 65% lower cost.

The framework grounds reward-hacking detection in a finite lifecycle model of an evaluation's reward-relevant events, combining a static phase-aware taint analysis with runtime infrastructure-side evidence attribution. Evaluation used a human-labeled corpus of 456 adjudicated trajectories drawn from more than 31,000 public agent runs across three benchmarks. BenchShield improves full-chain recall from 23-94% to 77-100% and same-vector coverage from 16-56% to 43-78%, cuts per-task cost by up to 65%, and achieves 96% accuracy detecting reward hacking at runtime.

arXiv cs.CR·Sep 10, 03:10 UTC · 6d agoAI safety & security1

42

When the prompt becomes the payload: A practical pen-testing guide for GenAI, LLM and RAG applications

CSO Online publishes a practical penetration-testing guide for GenAI, LLM, and RAG applications, covering prompt injection, retrieval poisoning, and tenant isolation testing.

The guide frames LLM applications as attack graphs spanning prompts, retrieval layers, vector stores, tools, identities, and downstream APIs, arguing that conventional web testing misses instruction-vs-data channel risks. It builds on OWASP prompt injection guidance (direct vs. indirect injection) and NIST's 2025 adversarial machine-learning taxonomy, noting that RAG and fine-tuning do not remove injection risk. Recommended practices include documenting trust transitions across components, using canaries and synthetic records to avoid test side effects, running multi-turn and obfuscated injection campaigns, and verifying chains from poisoned documents to observable state changes. It also details testing RAG pipelines via controlled document poisoning across metadata, OCR layers, and code comments, plus cross-tenant isolation checks on retrieved document IDs.

CSO Online·Sep 9, 10:00 UTC · 7d agoAI safety & security1

28

Your Agent Aced the Task. Will It Do It Again?

IBM Research Hugging Face post examines whether LLM agents that succeed at a task once will reliably succeed again.

Hugging Face published an IBM Research blog post titled 'Your Agent Aced the Task. Will It Do It Again?' with URL slug 'altk-evolve-consistency'. No article text was provided, but it appears to address agent consistency and reliability evaluation across repeated task runs. This is relevant to developers building or evaluating LLM agent systems.

Hugging Face Blog·Sep 15, 16:00 UTC · 1d agoAI tools & infra

30

Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

Embodied-BenchForge automates embodied benchmark construction via closed-loop synthesis with verification and repair, yielding seven benchmarks for MLLM evaluation.

Embodied-BenchForge is an agentic framework that transforms user-specified evaluation intents into complete embodied benchmark artifacts via Closed-Loop Benchmark Synthesis. Skill-Orchestrated Artifact Synthesis composes typed reusable skills while an artifact dependency graph records intermediate outputs; Requirement-Guided Verification and Repair triggers local re-execution or upstream rollback on failures. It constructs six Offline EQA benchmarks plus one interactive benchmark with 220 executable tasks, distinguishing MLLM and embodied agent capabilities in observation-based understanding and closed-loop execution.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:16 UTC · 5d agoAI research

18

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

VDiff-Bench, a 1,756-question benchmark, shows multimodal LLMs struggle with fine-grained image-difference identification, scoring as low as 8.7% on low-level changes.

VDiff-Bench is a multiple-choice benchmark of 1,756 four-way questions over image pairs covering 10 change categories including position, motion, color, texture, OCR/text and illumination, with curated hard negatives. Evaluation of 11 state-of-the-art open- and closed-source MLLMs shows fine-grained visual comparison remains brittle: 7-8B-scale open-source models score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture. Notably, Grok 4.3 shows a sharp performance drop on noise and texture differences, falling behind large open-source models like Kimi K2.5 and K3.

Hugging Face daily papers·Sep 4, 20:00 UTC · 12d agoAI research

32

xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

xDailyBench tests 11 frontier LLMs on 248 real-life consultation tasks; the best models score 75.6% and lag on implicit requirements.

The benchmark spans 51 scenarios across personal life, white-collar work, learning and research, and cross-domain activities, grounded in requests users actually completed or intended to complete with AI. Tasks are scored with fine-grained binary rubrics covering explicit and implicit requirements under standardized agentic settings. Across 11 frontier models, the best achieved a 75.6% task-level score, with all models performing at least 9 percentage points worse on implicit than explicit requirements.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:30 UTC · 9d agoAI research

26

τ^τ-Bench: An Environment for End-To-End, Realistic Agent Construction

New τ^τ-bench tasks coding agents with building deployable customer-service agents; best config, Claude Opus 5, passes only 23.9% of simulations.

Researchers introduce τ^τ-bench, an end-to-end benchmark where a developer agent must build a complete customer-service agent from real business records, a client with requirements, a production API, an inherited codebase, and cost/model limits, then is scored by deploying it against held-out simulated users. Across 53 tasks in four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations versus an 82.2% expert-authored reference ceiling. Failure modes mirror those of human developers: shallow queries instead of deep record comprehension, almost no client communication, and shipping the first architecture that runs rather than experimenting.

Hugging Face daily papers·Sep 3, 20:00 UTC · 13d agoAI research

50

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

A review paper frames on-policy self-distillation collapse as governed by three levers: token weighting, privileged information, and guidance decay.

The paper critically reviews On-Policy Self-Distillation (OPSD), where a language model trains on its own generations scored token-by-token by a teacher conditioned on privileged information such as reference solutions or environment feedback. It identifies collapse, the progressive narrowing of producible reasoning paths, as the dominant failure mode and analyzes it through three levers: signal weighting, the nature of privileged information, and teacher dynamics. The review is restricted to mathematical reasoning, reports no new experiments, and offers a shared vocabulary separating settled findings from disputed ones.

Hugging Face daily papers·Aug 25, 20:00 UTC · 22d agoAI research

45

AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

Open-source speech foundation model AuK unifies generation and editing, trained on 1.95 million hours, with distilled AuK-Flash achieving 4.5x speedup.

AuK is an open-source foundational model that unifies speech generation and editing through natural-language instructions and audio context, trained on approximately 3.03 billion instruction-audio instances and 1.95 million hours of supervision across five task families including generation, content editing, and acoustic editing. It combines a multimodal LLM for semantic conditioning, a VAE jointly trained on speech, general audio, and music, and a hybrid rectified-flow Transformer using dual-stream MMDiT blocks followed by unified single-stream DiT blocks. Post-training applies human-feedback preference optimization for editing and reward-based reinforcement learning for generation, and the distilled AuK-Flash performs 4-step inference without classifier-free guidance at a 4.5x wall-clock speedup. Source code and model weights are released.

Hugging Face daily papers·Sep 7, 20:00 UTC · 9d agoModel release2

42