Search: “reasoning”

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

Quesma benchmarks Qwen3.8 27B quantizations: 4-bit Q4_K_M matches BF16 on key benchmarks while 1-bit collapses to random chance.

Quesma spent roughly $3,000 on Modal GPUs testing Unsloth GGUF quantizations of Qwen3.8 27B across GPQA Diamond, IFBench, and Terminal-Bench 2.1. The 17 GB Q4_K_M quantization matched the 55 GB BF16 model on Terminal-Bench 2.1 and showed little degradation down to 4-bit, while the 2-bit UD-Q2_K_XL dropped noticeably. At 1-bit, scores on GPQA Diamond fell to random-guess levels, with longer reasoning making results worse, and reasoning effort settings significantly affected outcomes.

Hacker News · AI·Sep 8, 14:49 UTC · 8d agoAI researchHN 56↑ · 30 comments1

SpatialBlock: Enhancing Spatial Intelligence in LVLMs via Synthetic Block-Stacking Problem

Researchers release SpatialBlock-15k, a synthetic block-stacking dataset that improves 3D spatial reasoning in large vision-language models without dense geometric annotations.

The paper addresses limited spatial intelligence in LVLMs by training on structured block-manipulation tasks instead of costly real-scene annotated datasets. SpatialBlock-15k contains 15,000 synthetic problems covering 3D-to-2D projection, viewpoint transformation, and structural combination, with color modulation as visual cues. LVLMs trained on it via direct answering or reasoning-based prediction outperform baselines and generalize to real-world spatial tasks. Code and data are released on GitHub.

Hugging Face daily papers·Sep 6, 20:00 UTC · 10d agoAI research

Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models

Audit of 22 frontier models finds widespread verbatim retrieval of published molecular property values, with higher reasoning increasing recall of memorized numbers.

An arXiv audit tests 22 frontier LLMs across 12 molecular regression benchmarks for verbatim retrieval of published values. More than 50% of the LLMs show verbatim retrieval on five datasets, and identical experiments are flagged 89% more often at a high reasoning level than at the lowest one. Suppressing retrieval moves model prediction errors closer together in relative terms, suggesting predictive capability is not determined solely by memorized values.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:32 UTC · 12d agoAI research1

RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

RoboSPA introduces a 527K-trajectory robotic benchmark with 280 task variants exposing VLA model weaknesses in spatial reasoning and long-horizon planning.

RoboSPA is a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in vision-language-action models, spanning fine-grained spatial reasoning and long-horizon procedural planning. It covers 10 task categories and 56 base tasks instantiated across five difficulty levels for 280 variants, with 527K trajectories collected across multiple embodiments. Experiments on representative VLA models show current systems struggle with complex spatial relations, precise low-level execution, and memory-intensive planning.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 16:19 UTC · 12d agoAI research

RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

RoboSPA introduces a 527K-trajectory benchmark with 280 task variants showing current VLA models struggle with spatial reasoning and long-horizon planning.

RoboSPA is a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in vision-language-action models, covering fine-grained spatial reasoning and long-horizon procedural planning. It spans 10 task categories and 56 base tasks instantiated across five difficulty levels for 280 variants, with 527K trajectories collected across multiple embodiments and diverse scenes. Diagnostic metrics go beyond binary success rates, and experiments show current VLA models struggle with complex spatial relations, precise low-level execution, and memory-intensive planning.

Hugging Face daily papers·Sep 3, 20:00 UTC · 13d agoAI research

5 Reasons Device Management Isn't Device Trust​

The Hacker News·Apr 22, 04:37 UTC · Apr 22, 2025AI research

MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education

Introduces MUSE, a twelve-task benchmark evaluating vision-language models on artistic image understanding in situated educational, Southeast Asian contexts.

MUSE is a benchmark assessing large vision-language models on artistic image understanding across twelve tasks spanning visual perception, semantic and affective interpretation, cultural understanding, and compositional reasoning. It decouples image annotation from question generation for controllable difficulty and curates images centering Singaporean and Southeast Asian multicultural contexts alongside Western art. Evaluations of open-source and proprietary models found substantial disparities, especially in affective interpretation and compositional reasoning.

arXiv cs.AI / cs.LG / cs.CL·Sep 16, 17:26 UTC · 20h agoAI research

Bellman Policy Optimization

Bellman Policy Optimization, a critic-free RLVR method derived from Policy Mirror Descent, improves LLM mathematical reasoning without intermediate state-value estimation.

The paper introduces Bellman Policy Optimization (BPO), a critic-free reinforcement learning method for LLMs with verifiable rewards, derived from Policy Mirror Descent. BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective for autoregressive generation with terminal rewards, avoiding state-value estimation at intermediate states. The authors prove BPO shares the same unique optimal solution as PMD and validate it on mathematical reasoning benchmarks.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:59 UTC · 2d agoAI research

Agent as Policy for Robotic Manipulation

Agent as Policy lets a general-purpose agent drive a physical robot via runtime reasoning and program generation, reaching 100% success on manipulation tasks.

The paper introduces Agent as Policy (AGP), which puts task planning and execution for a physical robot under a general-purpose agent's control with no task-specific or environment-specific training. The agent interprets visual evidence, writes executable programs, issues motion commands, and revises actions based on physical outcomes. AGP was evaluated on real-world manipulation tasks including assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. It achieved success rates of 100%, 100%, and 80% on three block construction configurations.

Hugging Face daily papers·Sep 10, 20:00 UTC · 6d agoAI research

I tested 10 model/harness combinations on the same Three.js task

A developer benchmarked 10 model/harness combinations on a Three.js task; Qwen 3.8 27B on OpenCode scored 95.64% fastest at 8m48s.

The author ran an identical Three.js sci-fi hangar build prompt across 10 model/harness combinations and recorded score, tokens, durations, and tool errors. Qwen 3.8 27B x-high on OpenCode achieved 95.64% in 8m48s, the best fast result, while GLM 5.3 Flash Max on OpenCode scored highest at 96.89% in 20m28s. Other runs included GLM 5.3 Flash, Luna 5.6, SOL 5.6, and Astra 6.0 across Codex Open, OMP Open, OpenCode, DSH, and PTC harnesses, with scores ranging from 78.54% to 96.89%.

Hacker News · AI·Sep 8, 03:42 UTC · 9d agoAI researchHN 104↑ · 61 comments1

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs

SAFIRE, an 83K-image fire and smoke benchmark, shows open-source multimodal LLMs average only 61.9% accuracy on safety-critical fire reasoning.

SAFIRE is a large-scale benchmark for fire-smoke understanding in multimodal LLMs with 83K captioned images across 20 scenarios and 193K multiple-choice VQA questions spanning 10 evaluation dimensions from perception to higher-order reasoning. Annotations were built via a GPT-5.4-assisted multi-stage pipeline with MLLM majority voting. Ten open-source MLLMs (8B-38B) average 61.9% accuracy, exposing major gaps in safety-critical reasoning. Adapting vision encoders on 7% of the domain data raises fire-scene classification from 20.1% to 64.5%.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:59 UTC · 9d agoAI research

ReactVAU: A Slow-Fast Decoupled Framework for Streaming Video Anomaly Understanding

ReactVAU is a slow-fast streaming framework for real-time video anomaly understanding that reserves heavyweight MLLM reasoning for suspicious events, improving efficiency.

ReactVAU addresses causal streaming video anomaly understanding with three components: a lightweight Fast Detection Module using Spatial Grid Folding, Anomaly-Aware Persistent Memory that protects critical visual cues from temporal decay, and a Slow Reasoning Module activated only on suspicious events. This design minimizes heavyweight MLLM invocations during long normal intervals. Experiments show competitive anomaly detection and causal reasoning under strict streaming constraints with significantly enhanced computational efficiency.

Hugging Face daily papers·Sep 6, 20:00 UTC · 10d agoAI research

The Pelican comparison grid for Astra is pretty interesting

Simon Willison's pelican SVG comparison shows GPT-6 Astra producing markedly better images than GPT-5.6 Sol, Terra, and Luna across reasoning levels.

Willison generated pelicans-riding-bicycles SVGs with newly accessed GPT-6 Astra at low through max reasoning levels and rendered them in a comparison grid against GPT-5.6 Sol, Terra, and Luna. Astra's outputs were markedly more coherent, while even the best GPT-5.6-Sol images remained largely abstract shapes. Astra does not support a reasoning=none setting, so all comparisons involved reasoning-enabled runs.

Simon Willison·Sep 4, 23:59 UTC · 12d agoAI research

TempCloze: Can Video-LLMs Identify the Missing Middle?

TempCloze benchmark tests Video-LLMs' temporal reasoning with 1,521 videos, finding temporal alignment is the primary failure mode across 31 models.

TempCloze is a video cloze benchmark in which models must identify the true missing middle clip given the beginning and ending clips, using 1,521 carefully filtered videos from seven sources, mostly long-take and egocentric footage. Distractors are constructed along three dimensions: Semantic, Alignment and Progression, with shared scenes and objects to reduce appearance cues. Evaluation of 10 proprietary and 21 open-source Video-LLMs found Alignment is the primary bottleneck, with models often recognizing plausible semantics and local event progression but struggling with temporal alignment.

Hugging Face daily papers·Aug 31, 20:00 UTC · 16d agoAI research

AgenticGen: Reward-Guided Agentic Video Generation for Advertising

AgenticGen applies DPO and GRPO reward-guided reasoning to ad video generation, improving TikTok CTR 2.72%, CVR 2.63%, and Advv 9.61%.

AgenticGen decomposes advertising video generation into two trainable reasoning stages, strategy selection and draft generation, supervised by online business feedback. It learns a performance-based reward from accumulated online feedback plus a rubric-based reward aligned with human quality standards, then optimizes policies with DPO followed by GRPO using process and outcome rewards. Online A/B experiments in the TikTok advertising system show CTR up 2.72%, CVR up 2.63%, and Advv up 9.61% over an SFT baseline.

Hugging Face daily papers·Aug 30, 20:00 UTC · 17d agoAI research

Coaching AI agents: Why your next security hire might be an algorithm

Help Net Security·Sep 9, 09:54 UTC · Sep 9, 2025AI research

Putting Usability First: How to Design a Great User Experience

Recorded Future·Jun 27, 00:00 UTC · Jun 27, 2025AI research

Safe Meta-Reinforcement Learning via Information Space Reachability

Safe meta-RL framework reasons about safety in information space, learning a safety value function used for safety filtering and constrained policy optimization.

The paper proposes safe meta-RL that reasons about safety in information space, capturing both physical state and the agent's belief over the underlying task. A safety value function measures the probability of avoiding unsafe regions indefinitely and satisfies a self-consistency condition and Bellman equation, making it learnable via meta-RL. The resulting algorithm uses the learned function for safety filtering and constrained policy optimization, with effectiveness demonstrated on meta-RL benchmarks.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:27 UTC · 2d agoAI research

Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models

Drift-Constrained Optimization reformulates fine-tuning as update-direction selection, letting Qwen3 models improve target tasks within a behavioral drift budget.

The paper specifies a behavioral drift budget before optimization and shows that update direction is the remaining degree of freedom, reformulating fine-tuning as a direction-selection problem. In a stringent QA-only setting where instruct models must still generate multi-step reasoning at inference, a coarse layer-selective probe reverses the failure of QA-only fine-tuning. Across Qwen3-8B and Qwen3-14B, these directions substantially improve scientific reasoning and multilingual translation, matching or outperforming dedicated translation systems over 100+ languages and giving stronger initialization for reinforcement learning.

Hugging Face daily papers·Sep 11, 20:00 UTC · 5d agoAI research

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

SAS trains a gated sparse attention selector end-to-end with the language modeling loss, beating distillation-based sparsifiers on reasoning, long-context, and agentic tasks.

The paper proposes Simple Attention Sparsification (SAS), which injects a selector's continuous scores into attention softmax logits so the language modeling loss directly optimizes context ranking instead of distilling dense attention distributions. Key design choices include log-form gates inside the softmax, normalized gates calibrated against the current block, and preserved continuous selector scores. A memory-efficient Triton kernel integrates SAS into FlashAttention-style computation for long-sequence training. SAS outperforms trainable sparse attention baselines across budgets, with the largest gains under tight attention budgets.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:58 UTC · 5d agoAI research 2 sources2

JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

JarvisGUI benchmark tests GUI agents on cross-device workflows across Android, Windows, and Ubuntu, revealing major gaps in state transfer and long-horizon reasoning.

JarvisGUI is a dynamic benchmark that formulates GUI tasks as input-output transformations under a lightweight type system, automatically composing multi-step cross-device workflows across Android, Windows, and Ubuntu virtual environments. Evaluation shows state-of-the-art open-source GUI agents struggle with state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management, exposing a capability gap invisible to existing single-device benchmarks.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:59 UTC · 7d agoAI research1

Why AI food looks like that

Experts explain why AI-generated food images look unappetizing, citing diffusion model limitations, weak structural reasoning, and stylized training data.

The Verge examines why AI-generated food imagery from restaurants and brands often appears grotesque, citing researchers from Oxford, Naples, Zurich, and London. Diffusion models recover coarse structure before fine texture, so structural errors like extra fingers or donut shrimp get baked in early. Researchers note the models are weak at thin, continuous, terminating structures such as noodles, and reproduce the glossy conventions of professional food photography without understanding the objects. Odd internet imagery and memes in training data further skew outputs toward strange textures and clustered holes.

The Verge · AI·Sep 4, 11:00 UTC · 13d agoAI research

Lessons from the hacks

The recent run of cyberattacks by in-development frontier models has got me thinking a lot about how our current incentive systems are not well suited for such fast technological transitions. The two primary power structures here are the rapidly growing technology companies and the federal government. The companies are incentivized to grow, so they can keep growing and keep scaling – in what is…

Interconnects·Aug 9, 14:57 UTC · Aug 9, 2026AI research

Can Skills Learned in Games Transfer to Real-World Work?

Good Start Labs trains models in strategy games like 1830 and Diplomacy, showing terminal-agent training transfers to financial research benchmarks.

Good Start Labs, spun out of Every with $3.6M from General Catalyst and Inovia, trains AI models in verifiable strategy games. A 30B model trained as a multi-turn terminal agent in 1830: The Game of Railroads and Robber Barons improved Finance-Agent benchmark performance, while single-turn QA training did not transfer. The founders also co-authored COS-PLAY, a paper on co-evolving LLM decision and skill-bank agents for long-horizon tasks.

Latent Space·Sep 15, 20:11 UTC · 1d agoAI research

JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management

JustFit MLX runtime serves 200K-token contexts for Qwen3.8-27B on a 24 GiB MacBook via just-in-time state management.

JustFit is an MLX-based inference runtime combining KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions, independent of weight quantization. On a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, it completed 196,608 input and 16,384 output tokens, raising single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x). Performance tests show 19.11 tokens/s on a 32K-input probe with a 16,374 MiB median peak footprint, and the runtime answered 29 of 30 AIME 2026 problems correctly.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:15 UTC · 1d agoAI research

Agent Harness vs Agent Framework vs MCP: Which Layer Owns the Loop, State, Tools, Permissions, and Recovery

Architecture explainer separates agent harnesses, frameworks, and MCP by which layer owns the loop, state, permissions, and recovery.

The article distinguishes agent harnesses (OpenAI Codex, Claude Agent SDK), which own the execution loop, sandbox, permission model, and recovery; frameworks (LangGraph, OpenAI Agents SDK, Microsoft Agent Framework), which supply composable primitives; and MCP, a stateless JSON-RPC wire protocol governed by the Linux Foundation's Agentic AI Foundation since December 2025. An ownership matrix maps the execution loop, state, tool transport, permissions, recovery, sandboxing, and multi-agent orchestration to each layer. The 2026-07-28 MCP specification made the protocol fully stateless, retiring the initialize handshake and session headers.

MarkTechPost·Sep 14, 21:54 UTC · 2d agoAI research1

Discovery Foundation Models: Toward Open-Ended Discovery Intelligence

Proposes Discovery Foundation Models that participate in creating new problems and knowledge, instantiated in Zetema and the GALILEO therapeutic-discovery system.

The paper formulates Discovery Foundation Models (DFMs) as general-purpose systems for open-ended discovery, supporting seven coupled capabilities from problem discovery through evidence-grounded revision and continual improvement. It instantiates the framework with Zetema, combining explicit research-state dynamics, verification gating, external grounding, and cross-task Discovery Skill evolution. GALILEO, a real therapeutic-discovery system, closes the loop between Dry-Lab reasoning, robotic and hands-on Wet-Lab experimentation, and iterative hypothesis revision. The authors also define process-centered evaluation so discovery behavior can be trained and measured beyond final answers.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:56 UTC · 2d agoAI research 2 sources

Before You Poll with LLMs: A Deliberative Diagnostic Framework

Deliberative diagnostic shows all five tested frontier LLMs misrepresent human belief shifts after arguments, with GPT-5.1 reversing on outgroup questions.

The Deliberative Polling Diagnostic Framework compares human and LLM persona belief shifts after identical informational interventions, using data from America in One Room (526 personas, 72 questions). All five frontier models tested failed uniquely: GPT-5.1 exhibited partisan reversal (80% on outgroup vs 26% on policy questions), Gemini 2.0 Flash, Claude Sonnet 4.5 and Llama 3.3 70B overshot at 5-7x human magnitude, and DeepSeek V3 showed near-zero change (rigidity). The authors term the underlying signature 'self-sycophancy', conformity to the model's internal persona stereotype rather than reasoning from provided information.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 16:41 UTC · 2d agoAI research

Why don't machine learning research agents overfit?

Amazon researchers explain why ML research agents avoid benchmark overfitting, attributing generalization to compressibility of successful strategies.

Amazon Science summarizes the paper "What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents," which investigates why benchmark hill-climbing loops, whether run by human communities or LLM research agents, do not produce rampant overfitting. The explanation formalizes Occam's razor via a counting argument: successful ML strategies are highly compressible, so short descriptions lack room to memorize benchmark data and must capture real structure. LLM-based agents, being resettable and controllable, allow this hypothesis to be tested empirically.

Hacker News · security·Sep 14, 16:32 UTC · 2d agoAI researchHN 21↑ · 3 comments

Claude Fable 5.1 Solves the Cyphral Distich, a 370-year-old cipher

Claude Fable 5.1 solved Sir Thomas Urquhart's 370-year-old Cyphral Distich cipher, recovering a hidden royalist prayer for Charles II.

Vals AI reports that Claude Fable 5.1 solved the Cyphral Distich, a 64-number cryptogram from Sir Thomas Urquhart's Logopandecteision unsolved since 1653, in 44 minutes using 176k tokens with no human hints. The key insight was that the cipher's key was the book itself: each number indexes a word in the corresponding Proquiritation, taking the first letter, yielding 'O GOD UPHOLD KING CHARLS THE SECOND AND MAKE HIM THE SUPREME RULER OF THIS LAND'. The model also deciphered the larger Cyphral Octastich (285 numbers) from The Jewel (1652) using page-based word indexing, recovering all but nine letters of a royalist prayer. The puzzle had been listed among Klaus Schmeh's Top 50 unsolved encrypted messages.

Hacker News · AI·Sep 13, 21:06 UTC · 3d agoAI researchHN 63↑ · 6 comments1· 1 read

Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States

Mind2Dialogue simulates users' mental states to generate privileged supervision, boosting personalization and preference-following in Qwen, Llama, and OLMo assistants.

The Mind2Dialogue framework uses a psychology-guided simulator that preserves personal characteristics while updating user mental states through interaction, driving coherent conversations and an Oracle assistant's responses. Privileged distillation trains models on the Oracle's well-informed responses so they can assist users without direct access to mental states at deployment. Training on the full corpus improves every reported personalization metric over Qwen, Llama, and OLMo instruction-tuned baselines, including 26.6 to 40.9 percentage point gains in preference-following generation.

Hugging Face daily papers·updated Sep 14, 17:55 UTC · 2d ago·first Sep 13, 20:00 UTC · 3d agoAI research 2 sources1

BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender

Blender-VideoBench evaluates agentic video understanding by having agents programmatically reconstruct real videos in Blender scenes.

BVB (Blender-VideoBench) tests whether multimodal agents truly understand videos by requiring programmatic reconstruction of real-world videos as animated Blender scenes via a lightweight Mini-BVB harness under identical sandbox and cost constraints. Evaluation uses Dual VQA for spatiotemporal fact preservation and Latent Similarity for perceptual match, combined in a square-root mean overall score. Across 51 configurations from 10 model families, the best model reaches 88.6 Latent Similarity but retains only 53.7% of source-correct spatiotemporal answers, showing semantic retention remains the main challenge.

Hugging Face daily papers·Sep 13, 20:00 UTC · 3d agoAI research

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

Real-SWE benchmark tests coding agents on licensed private enterprise codebases; top model Fable 5.1 resolves only 38.8% of tasks.

Real-SWE is a new benchmark evaluating frontier AI coding agents on tasks drawn from private production codebases licensed from real companies, spanning billing, tax calculation, and cross-service migrations. Fable 5.1 with Claude Code leads at 38.8% resolution rate (pass@1 over eight runs), followed by GPT-6 Astra Codex CLI at 33.8% and Gemini 3.8 Flash Gemini CLI at 31.2%. Tasks use native harnesses and realistic tooling including Docker, Kubernetes, PostgreSQL, Redis, and Linear; median reference solutions edit 11 files versus 6 for DeepSWE and FrontierCode.

Hacker News · AI·Sep 12, 20:25 UTC · 4d agoAI researchHN 45↑ · 31 comments1

AI agents are flooding public services with new requests

Researcher documents 'agentic flooding' across 84 cases in 11 jurisdictions as AI tools drive surging complaint volumes at public services worldwide.

TechCrunch covers researcher Chris Schmitz's paper documenting 'agentic flooding' across 84 potential cases in 11 jurisdictions, where AI tools drive surges in filings to public services. UK housing ombudsman complaints rose from 2,600 in 2022 to over 7,000, and CFPB complaints grew fivefold over the same period, with similar jumps in Brazilian and German petitions. The paper, set for presentation at the AI Ethics and Society conference, argues most new filings are legitimate claims previously blocked by administrative burden, and its dataset is publicly released.

TechCrunch · AI·Sep 10, 14:53 UTC · 6d agoAI research

Feature Recovery for Object Understanding After Irreversible Fire Damage

TRACE benchmark with 21.4K scenes studies post-fire object understanding; a Feature Recovery Module improves degraded-image retrieval by 12.5% and material recovery by 20.1%.

The paper introduces TRACE, a transformation-aware benchmark with 21.4K real-image-grounded synthetic scenes, 499 object identities across 189 categories, and five tasks covering degraded-object detection, pristine-state recovery, material recovery, description generation, and functional reasoning. Existing models degrade sharply: RF-DETR mAP falls 71% relative from least to most severe level, and InternVL3.5 retrieval R@1 drops from 93.85 to 28.11. The proposed Feature Recovery Module maps degraded encoder features to pristine-aligned representations while keeping the host model frozen, averaging relative gains of 12.5% for retrieval and 20.1% for material recovery across VLM hosts and severity levels.

Hugging Face daily papers·Sep 9, 20:00 UTC · 7d agoAI research

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

MetroLLM-Bench is a 955-case benchmark testing language models as transit kiosk tool-calling runtimes across six real metro systems.

The benchmark covers 37-414-station metro systems and eleven task categories including routing, fare calculation, disruptions, accessibility, and adversarial input, with 14 deterministic and 8 semantic scoring components. Of 26 models from six vendors, a PEFT-tuned 4B Qwen 3.5 student scored 91.3 on Tier 1, exceeding GPT-5.6 (90.6/90.0), while Muse Glimmer 30B led the composite ranking. A deterministic rule-based baseline reached 84.6, and PEFT gains over base models shrank from +7.03 points at 2B to -0.91 at 27B.

Hugging Face daily papers·Sep 8, 20:00 UTC · 8d agoAI research1

Latest open artifacts (#24): Motif-3, GLM-5.3, Hy4-preview and open model licenses

Interconnects surveys new open models—Motif-3, GLM-5.3, Hy4-preview—while analyzing a licensing split: Western labs opening up, Chinese frontier labs getting restrictive.

The roundup covers Motif-3 (MIT license, strong scores for its size), GLM-5.3 (switched from MIT to a custom license with a $10 billion revenue threshold and undefined 'affiliates' clause requiring Z.AI security review), and Tencent's Hy4-preview (competent but prone to overthinking). It also notes dots3-note-prev from RedNote/Xiaohongshu (won IMO 2026 with a perfect score), Qwen3.8-Flash-Next (125B-A6B with GDN and Qwen Sparse Attention), NVIDIA Nemotron-3.5-Lightning-30B-A3B-BF16, and Ling-3.0-flash. The core theme: Google and Meta adopted Apache 2.0 while Chinese frontier labs (Zhipu, Kimi K3, MiniMax M3) adopted restrictive commercial licenses.

Interconnects·Sep 8, 14:15 UTC · 8d agoAI research

Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber Models

A seven-person independent team trained open-weight agentic cyber models via a data-centric post-training framework, ranking 10th on CyberGym and first at comparable scale.

The paper presents Feyospace-v1, a data-centric post-training framework combining five systems: Choulea (hidden reasoning analysis), SkyReal (teacher-sampling cost reduction), Hongzwang (bypassing teacher API restrictions), PSBreakup (restoring capabilities weakened by model merging), and Kreator (converting expert interventions into trainable reasoning). The data engine builds resettable coding, vulnerability, CTF, kernel-history, full-exploit, firmware, and device-backed environments, retaining only execution-verified and evidence-audited trajectories, yielding 164,269 trajectories for long-context supervised fine-tuning. Three checkpoints improve over their starting models by an average of 23.76% on the full CyberGym suite and 10.49% across pooled CTF suites. As of September 1, 2026, Feyospace-s1 achieves a 63.24% verified success rate, ranks 10th on the official CyberGym leaderboard, and all three checkpoints rank 1st among models at comparable parameter scales.

Hugging Face daily papers·Sep 7, 20:00 UTC · 9d agoAI research

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

A review paper frames on-policy self-distillation collapse as governed by three levers: token weighting, privileged information, and guidance decay.

The paper critically reviews On-Policy Self-Distillation (OPSD), where a language model trains on its own generations scored token-by-token by a teacher conditioned on privileged information such as reference solutions or environment feedback. It identifies collapse, the progressive narrowing of producible reasoning paths, as the dominant failure mode and analyzes it through three levers: signal weighting, the nature of privileged information, and teacher dynamics. The review is restricted to mathematical reasoning, reports no new experiments, and offers a shared vocabulary separating settled findings from disputed ones.

Hugging Face daily papers·Aug 25, 20:00 UTC · 22d agoAI research

Import AI 469: Science AI; RSI simulator; and Zuck's technological pessimism

New DiG-bench benchmark of 70 hidden-rule games shows only Opus 5 and Fable 5 solving the hardest tiers, probing AI discovery and creativity.

Import AI 469 highlights DiG-bench (Discovery in Games), a benchmark of 70 handcrafted games with hidden rules and objectives where only 21 games are public and most are kept private to avoid training contamination. Only Opus 5 and Fable 5 with Claude Code solved any Tier 7 tasks (about 0.2 success), with GPT-5.5 next; the games are text-based and have beaten every human tester at least once. The newsletter also covers an RSI simulator game by Paradigm Research and Inherent's Faraday, a post-trained open-weight model that supervises frontier models to improve scientific research output.

Import AI·Aug 17, 13:05 UTC · Aug 17, 2026AI research