Search: “calibration”

A Ranking Approach for Measuring Calibration

Researchers propose rankECE, a ranking-based calibration error measure with theoretical guarantees that outperforms binned ECE approximations.

The paper introduces rankECE, an alternative to Expected Calibration Error (ECE) that measures miscalibration by comparing points with neighboring predicted-probability values. It addresses the impossibility of estimating ECE with guaranteed accuracy in assumption-free settings. Theoretical guarantees and empirical results establish rankECE as a better proxy for ECE than the binned approximations most commonly used in practice.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:34 UTC · 5d agoAI research1

SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators

SyncWorld is an action-conditioned world model acting as a zero-shot robotics simulator across unseen environments via visual calibration.

Researchers propose SyncWorld, an action-conditioned world model that simulates robot action outcomes in unseen environments without additional training. It uses a visual calibration episode of paired frames and actions to establish the setup-specific Action-Visual Mapping in context. Experiments show accurate simulation of action outcomes in novel settings and that simulated rollouts enable test-time policy improvement without training.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

Clinician-calibrated K-Bench evaluates 125 LLM configurations on 200 high-risk mental health vignettes, exposing wide variation in suicide and violence risk handling.

K-Bench is a clinician-calibrated, protected benchmark evaluating 125 model configurations from 33 base models across 14 providers on 200 multi-turn vignettes covering suicide, self-harm, domestic violence, substance misuse and no-risk presentations. A frozen GPT-4o judge achieved 94.2% exact agreement with clinician consensus across 6,751 eligible comparisons from 151 clinician-rated transcripts. Leading models combined supportive conversation with combined-risk scores above 95, while risk exploration varied substantially among weaker configurations; therapeutic prompting helped weaker models and elevated reasoning produced no average improvement. A continuously updated public leaderboard is hosted at k-bench.ai with protected test materials.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 16:49 UTC · 2d agoAI safety & security

Domain-Specific Hallucination Detection in Large Language Models

A multi-signal pipeline detects LLM hallucinations, reaching F1 0.915 on HaluEval and cutting Qwen2.5-0.5B hallucination rates from 85.5% to 37.7% via DPO.

The paper presents a hallucination detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo Dropout uncertainty, and temperature-scaled calibration. It achieves F1 0.915 and AUROC 0.977 on general-domain HaluEval tasks, with MC Dropout inference raising accuracy to 93.2%. Applying DPO to a Qwen2.5-0.5B generator reduces its hallucination rate from 85.5% to 37.7%, while cross-domain evaluation shows poor general-domain transfer to SciFact (F1 0.52) and PubMedBERT fine-tuning as the strongest adaptation (F1 0.63).

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:45 UTC · 6d agoAI research1

Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuning

CoTS temperature scaling cuts test-time prompt tuning's expected calibration error from 11.90% to 5.38% on ImageNet variants while raising accuracy.

The paper proposes CoTS, a post-hoc calibration method that applies temperature scaling to minimize the confidence gap between test-time-adapted and zero-shot predictions. A weak-strong ensemble variant, E-CoTS, further exploits multiple test-time augmentations to boost accuracy while maintaining calibration. E-CoTS reduces average expected calibration error from 11.90% to 5.38% on ImageNet variants while increasing accuracy from 60.74% to 62.95%.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 16:25 UTC · 1d agoAI research

Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback

CCL couples teacher calibration with student updates via token-level branching, provably removing teacher bias in LLM distillation.

The paper proposes Coupled Calibration and Learning (CCL), an LLM distillation algorithm that alternates teacher calibration using source-question reward feedback with student training on target questions under covariate shift. Each iteration calibrates the teacher on source feedback, trains the student on target questions, and lets the updated student inform subsequent calibration. The authors prove the student's expected KL divergence to the oracle student converges to zero at a polynomial rate, and show regularized direct matching error can remain bounded away from zero.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:15 UTC · 1d agoAI research

Jev: New frontier model 40-400x cheaper and 20-200x faster

TypeSafe AI launches Jev, an early-access 'System One' model delivering calibrated structured outputs claimed 40-400x faster and cheaper than LLMs.

TypeSafe AI, founded by former OpenAI researcher Diogo Almeida, released its first 'System One Model' called Jev in early access. Jev forgoes string generation and is trained with Reinforcement Learning for Calibrated Decisions (RLCD) to produce type-safe structured values with calibrated probabilities. The company claims 70-500ms response times (40-200x faster), input pricing of $0.042 per million tokens, and free output tokens via a parallel sampling architecture. Target use cases include AI-powered workflows, real-time applications, and verification/guardrail tasks.

Hacker News · security·Sep 15, 19:25 UTC · 23h agoModel releaseHN 59↑ · 11 comments

How GPT-5.6 Sol helps run quantum computing experiments

OpenAI describes an MIT researcher using GPT-5.6 Sol with Codex to autonomously run and calibrate quantum computing experiments.

OpenAI published a case study showing how an MIT researcher uses GPT-5.6 Sol together with Codex to autonomously run quantum computing experiments, analyze results, and calibrate qubits. The post is a product application story rather than a new benchmark, paper, or model release.

OpenAI News·Sep 8, 17:00 UTC · 8d agoAI industry1

TransNormal-2: Geometry-Grounded Rectified Flow with Edge-Aware Decoding for Precise Normal Estimation

TransNormal-2 improves monocular surface-normal estimation by fixing VAE edge degradation with geometry-aware losses and refinement, matching MoGe-2 with 1.4% of annotations.

TransNormal-2 is a FLUX.2-based rectified-flow framework for monocular surface-normal estimation with single-step deterministic inference. The authors quantify that VAE 8x spatial compression introduces 1.3-8.5 degrees of mean angular error even on ground-truth normals, with edge error up to 2.8x the global error. The method adds geometry-aware pixel-space losses and an RGB-guided Geometric Refinement Module to correct boundary-localized decoding errors. It matches or exceeds MoGe-2 on all eight reported metrics using only 1.4% as many task-specific annotations, and cuts transparent-object MAE by 4.2 degrees on ClearGrasp and 3.1 degrees on ClearPose.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

Building a Production Greek-English Speech Recognizer

Engineering report details Sophea, a production Greek-English ASR reaching 4.26% WER on public English sets via ROVER ensemble and data-pipeline calibration.

Across 23 training iterations, two architectures, and nine production gates, no single data composition passed all gates; a three-model ROVER ensemble reached 9 of 9 gates and cut overlapping-speech WER from 53.35% to 37.87%. Calibrating an audio-quality filter against in-domain anchors reduced discarded scored Greek audio from 98.7% to 10.6%, and a pre-registered ablation traced a hallucination defect to one training-data package. The sophea/asr-k1 preview arbiter lists 4.26% average WER on eight public English test sets and 25.88% WER on live Greek noisy traffic; no weights or training data are released.

Hugging Face daily papers·Sep 10, 20:00 UTC · 5d agoAI research

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.

Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:57 UTC · 8d agoAI research1

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

PLC-DPO corrects noisy preference labels during DPO by routing each pair as clean, flip, or tie using calibrated policy-reference margins.

PLC-DPO addresses reversed, weak, or ambiguous preference labels in Direct Preference Optimization by using the calibrated policy-reference margin as online evidence to route each pair's training signal as clean, flip, or tie. This actively corrects supervision direction and strength rather than only filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO achieves the best mean win rate of 60.5 versus 55.5 for the next-best method. Injected-noise and tie stress tests plus human disagreement analysis show the routing stays stable and distinguishes flipped from weakly directional pairs.

Hugging Face daily papers·Aug 30, 20:00 UTC · 16d agoAI research

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled autoregressive testbed shows validation losses must be analyzed per task, and image tokenizer choice affects joint multimodal text modeling.

Researchers built a pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models, tracking task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They found that losses exhibit distinct scaling behavior per task and rank tokenizers differently, and that I2T loss over a shared text vocabulary gives a more consistent loss–performance signal than T2I loss. Better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and tokenizer choice can affect text modeling under joint optimization. Case studies examine the discriminator, semantic supervision, and vocabulary size design axes.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research1

Understanding Operator Attitudes Toward AI-Supported Decision Making in Maritime Operations

A survey of maritime professionals finds positive but scenario-sensitive attitudes toward AI collision-avoidance assistants, urging design for calibrated reliance.

The study surveyed maritime stakeholders on attitudes toward AI-supported collision-avoidance assistants for Maritime Autonomous Surface Ships, measuring technology anxiety, trust in automation, and explanation quality with established questionnaires plus thematic analysis of open responses. Results show generally positive disposition, no clear age-related openness differences, stable trust across scenarios, and multidimensional, scenario-sensitive explanation ratings. Participants valued decision support and situation awareness but worried about AI reliability, over-reliance, and skill loss; the authors recommend designing for calibrated reliance with domain experts in the loop rather than maximizing automation or trust.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 16:57 UTC · 6d agoAI research1

Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models

Audits of 10 classifiers on BRFSS show target leakage, not model class, drives the reported 0.89 AUROC in survey-based cardiovascular screening.

The study benchmarks ten model classes, including glass-box and tabular foundation models, for prevalent myocardial infarction on 442,067 respondents of the 2022 BRFSS across five feature tiers of decreasing leakage risk. Removing two post-diagnostic features costs every model 0.049-0.051 AUROC and collapses performance into a 0.0045-wide band, and the explainable boosting machine matches all alternatives within 0.005 while scoring roughly 104x faster than the strongest foundation model. Frozen models transport within 0.002 AUROC to 2023 data; the authors conclude evaluation practice and feature sets, not model capacity, are the binding constraint.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:29 UTC · 6d agoAI research1

Steering Geometry: Validating Human Value Geometry in LLM Steering Space

Researchers show LLM activation steering vectors encode human value geometry predicted by Schwartz's theory, with fidelity improving at scale but degrading after instruction tuning.

The study tests whether latent geometry of activation steering vectors reflects theory-specified structure in human values, using Schwartz's Theory of Basic Human Values and a new 26K-sample benchmark spanning 20 values. Distribution-driven methods (CAA, SphericalSteer, ODESteer) recover predicted value topologies with Spearman correlation up to 0.51 (p < 10^-13), while behavior-centric methods (COLD-Steer, BiPO) steer comparably but show little geometric fidelity. Geometric alignment improves with model scale but drops after instruction tuning. Better geometric alignment also yields more human-consistent cross-value transfer, steering one value lifting compatible values and suppressing opposing ones.

Hugging Face daily papers·Sep 4, 20:00 UTC · 11d agoAI research

Differentially Private EEG Feature Anonymization: A Privacy-Utility Case Study in Clinical Neurophysiology

Case study applies Gaussian and Laplace differential privacy to clinical EEG features, quantifying privacy-utility trade-offs across three deployment scenarios.

Researchers evaluate subject-level differential privacy for EEG-derived feature representations using Gaussian and Laplace perturbations across client-side, centralized server-side, and decentralized local training scenarios. Utility is assessed with statistical measures and a downstream machine-learning check on clinical neurophysiology data. Results show DP can be integrated into EEG workflows, but mechanism choice, privacy parameters, and sensitivity calibration strongly influence data utility, particularly on small and imbalanced clinical datasets. The study highlights the privacy-utility trade-off in protecting biomedical signals against re-identification and inference risks.

arXiv cs.CR·Sep 10, 16:31 UTC · 6d agoResearch

Measuring benchmark optimization in speech recognition

Hugging Face examines how much speech recognition systems overfit benchmarks and how to measure benchmark optimization in ASR.

A Hugging Face post on measuring benchmark optimization in automatic speech recognition, analyzing how model improvements on benchmarks reflect genuine capability gains versus overfitting. It is evaluation methodology research with no direct security impact.

Hugging Face Blog·Aug 21, 00:00 UTC · 26d agoAI research

nvidia/Qwen3.8-Flash-Next-NVFP4 — new model trending #28 on Hugging Face

NVIDIA released an NVFP4 4-bit quantized build of Alibaba's Qwen3.8-Flash-Next, a 125B-parameter MoE vision-language model, via Model Optimizer.

The checkpoint quantizes Qwen3.8-Flash-Next — a hybrid-attention (Gated DeltaNet and Qwen Sparse Attention) Mixture-of-Experts model with 125B total and 6B activated parameters, plus 51B n-gram embeddings and 4B MTP — using NVIDIA Model Optimizer v0.46.0. NVFP4 benchmarks stay close to FP8: GPQA Diamond 91.5 vs 92.0, MMMU Pro 78.3 vs 77.1, Terminal-Bench 2.1 82.9 vs 83.3. It targets Blackwell B200/B300 GPUs, runs on vLLM, supports 262K context extendable to 1M tokens, and is licensed under the NVIDIA Open Model License with Qwen Community License 1.0.

Hugging Face trending models·Sep 2, 04:03 UTC · 14d agoModel release

What Breaks Local Watermarks? A Robustness Benchmark for Local Invisible Image Watermarking

First systematic robustness benchmark of five local invisible image watermarking methods across 55 transformations finds all are vulnerable, with inpainting and geometric misalignment completely breaking payload…

The paper presents the first systematic robustness benchmark for local invisible image watermarks, covering 55 image transformations across signal distortions, coordinate alignment changes, indirect local edits, and direct watermark edits. It evaluates five methods: MaskWM, WAM, OmniGuard, TrustMark, and PixelSeal, all supporting localization natively or with minimal adaptation. Results show every method is vulnerable to some transformation; MaskWM offers the strongest payload recovery and localization but the lowest clean-image quality, and synchronization further improves its recovery under geometric transformations. Geometric misalignment and generative local edits such as inpainting and outpainting can completely impair payload recovery, while signal distortions are often tolerated.

arXiv cs.CR·Sep 15, 08:30 UTC · 1d agoResearch

Approval Integrity and Recovery in LLM Answer Publication

Study measures approval integrity in Lightcap LLM answer publication, finding the 14B response-act checker accepts 291 of 302 unsupported answers.

The study evaluates exact-content binding, authorization freshness, and checkpoint recovery in Lightcap's publication enforcement using 3,600 assessments over 900 human-annotated RAGTruth responses from three Ministral models. The production 14B response-act checker accepts 291 of 302 unsupported answers versus 41 for a direct-grounding baseline, with supported-answer retention of 95.2% versus 66.9%. A stateful recheck-recovery policy increases exact-match error by 9.23 percentage points relative to initial checkpoints, and controlled evidence-fingerprint changes expose asymmetric freshness enforcement between publication and recovery. A separate BIPIA prompt-injection experiment records zero target insertions among 266 valid editor outputs.

arXiv cs.CR·Sep 14, 13:48 UTC · 2d agoAI safety & security

Real-World Knowledge-Guided Change Data Synthesis for Remote Sensing

Researchers introduce KnowChange, a framework that uses pretrained vision-language models to synthesize realistic change-detection training data for remote sensing.

KnowChange is a knowledge-guided change data synthesis framework that leverages pretrained vision-language models to reason about plausible change locations and class transitions from pre-change scenes and desired change types. It addresses the limited class-transition coverage and inflexibility of handcrafted rule-based synthesis methods, enabling diverse change types in a unified pipeline. Experiments show KnowChange-generated data outperforms existing synthetic datasets in both synthetic-to-real transfer and synthetic data augmentation, despite compact generation scale.

Hugging Face daily papers·Aug 24, 20:00 UTC · 22d agoAI research

Illusion of Depth: Revealing Hidden Stereo Vision Vulnerabilities in Depth Estimation

Simple repeating patterns let attackers shift stereo-camera depth estimates by up to 20 meters, triggering emergency braking in autonomous driving frameworks at 40 km/h.

The paper reveals an intrinsic vulnerability in stereo cameras stemming from pixel sampling and calibration processes, letting attackers finely control estimated depth of real obstacles using simple repeating patterns without adversarial ML techniques. The attack was evaluated against BM and SGBM stereo matching algorithms, deep learning models PSMNet, MoCha-Stereo, and UniMatch, the stereo-LiDAR fusion model SGM-DDC, and commercial cameras ZED2 and Intel RealSense D435; on ZED2, obstacles can be displaced up to 20 meters farther or 12 meters closer. A 0.5-second attack triggered emergency braking in a popular autonomous driving framework, with feasibility confirmed at driving speeds up to 40 km/h using CARLA. State-of-the-art defenses proved ineffective, and the authors propose a similarity-score strategy to dynamically detect and suppress depth discrepancies.

arXiv cs.CR·Sep 14, 20:51 UTC · 1d agoResearch

ENCP: Episode-Normalized Conformal Prediction for Vision-and-Language Navigation

ENCP calibrates conformal prediction per navigation episode, giving step-level coverage guarantees for vision-language navigation agents despite within-episode dependence.

Episode-Normalized Conformal Prediction (ENCP) rescales a nonconformity score by a VLN policy's residual confidence and calibrates one maximum score per episode, preserving step-level coverage of at least 1−α despite dependence among steps within an episode. Across four VLN policies and three nonconformity scores on R2R and REVERIE, ENCP meets all reported empirical step-coverage targets in seen-to-unseen evaluation. The model-agnostic uncertainty estimates can signal when an agent should defer to a stronger predictor or human assistance.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:42 UTC · 1d agoAI research

Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesics

Researchers model curriculum learning as Wasserstein transport over difficulty distributions, finding curriculum benefits are strongly task- and budget-dependent with no dominant strategy.

The framework represents curricula as trajectories of training distributions over discrete difficulty levels, decoupling ordering, matched exposure, endpoint smoothness, and pacing. Across a calibrated suite of 12 tasks and 33 difficulty axes under fixed training budgets, no single strategy dominates, though easy-to-hard ordering improves hard-level performance relative to exposure-matched static sampling. Endpoint smoothness and pacing substantially affect where along the difficulty spectrum a curriculum is effective, and the transport view supports extensions to learned pacing and structured difficulty spaces.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:39 UTC · 8d agoAI research1

A Chosen Future Can Still Be Rewritten: Causal Writability in Video Models

Study shows video models often learn correct physics but fail to use it; low-dimensional 'causal writability' edits can restore correct motion.

The paper demonstrates 'causal writability' in video generation models: physically correct motion remains available inside the model even when the model outputs incorrect motion. In a red/blue mass oscillation setup, a low-dimensional edit predicted from simple physical variables restores correct fast motion, with a sharp depth boundary marking commitment. Early causal writability predicts which training errors later get corrected, and both writability and closure reproduce in a pretrained 1.3B video model.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:58 UTC · 2d agoAI research

When Intelligence Becomes Agency: A Theory of Governed, Proactive Agency for Symbiotic AI Systems

Researchers propose a formal framework for governed proactive AI agency, linking activation decisions to continuing authorization, accountability, and bounded delegation in symbiotic systems.

The paper defines the activation problem: persistent AI assistants must decide whether a situation warrants behavior, and whether to act, ask, monitor, defer, or deliberately refrain. It distinguishes autonomous from delegated agency and defines symbiotic agency as delegation under a standing, revocable mandate, coupled to the principal's situation with calibrated inference and bounded personalization. The framework links activation decisions to authorized perception, behavior selection, authority containment, traceable restraint, and constrained adaptation, and provides an agency classification method, evaluation framework, benchmark scenarios, and reference architecture for always-present assistants and embodied support systems.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 16:46 UTC · 9d agoAI safety & security

Attention Quantization for Tabular Foundation Models

FP8 quantization of attention queries, keys, and values speeds tabular foundation model inference up to 1.7x with no accuracy loss.

The paper develops an FP8 quantization strategy targeting attention calculations (queries, keys, values) in tabular foundation models, arguing attention matters more than weight or KV cache quantization given their differing size and serving patterns versus LLMs. Aligning quantization error between test rows and training rows proves crucial, since misalignment causes drastic accuracy drops. A Triton kernel using explicit FP8 matrix multiplication achieves up to 1.7x speedup over regular 16-bit kernels, with no relevant accuracy loss on TabPFN-v3 and TabICLv2 across TabArena and BeyondArena benchmarks.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:31 UTC · 5d agoAI research1

Fresh-Challenge VDF Attestations for Model-Relative Response Latency

Fresh-Challenge VDF Attestations bind verifiable delay functions to unpredictable public challenges, yielding succinct evidence of model-relative response latency.

The paper specifies FCLA, a protocol composition that binds a VDF to an unpredictable public challenge, a message, and independently auditable release/receipt records. Under explicit assumptions about VDF sequentiality and a calibrated bound on an adversary's sequential evaluation rate, an accepted transcript is inconsistent with post-challenge generation. A benchmark of the public reference implementation confirms the expected evaluation-versus-verification separation on one documented machine. The contribution is a protocol design analysis, not a new VDF construction.

arXiv cs.CR·Sep 11, 11:25 UTC · 5d agoResearch

Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation

Referee-Based Quality Estimation flags unreliable polyp segmentations at inference without ground truth, reaching ROC-AUC 0.960 with SegFormer-B0 referees.

RBQE measures agreement between a primary segmentation model and an independently trained referee on a 1,223-image external benchmark drawn from four public datasets. A cross-architecture SegFormer-B0 referee achieves the strongest signal (ROC-AUC 0.960), beating a Test-Time Augmentation baseline by 0.055 ROC-AUC under an identical protocol. Excluding trivially separable empty-mask cases, ROC-AUC falls to 0.876 (SegFormer-B0) and 0.783 (same-architecture control), but RBQE's margin over baselines widens. Progressive rejection of low-agreement predictions increases mean Dice of retained outputs, supporting selective prediction at the cost of one extra forward pass.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 17:32 UTC · 7d agoAI research

TART: A Modular Tool for Technique-Aware Audio-to-Tablature Guitar Transcription

TART pipeline transcribes guitar audio to tablature with fingering and technique labels, outperforming prior baselines by up to 8.5 Tab F1 points.

TART is a modular four-stage pipeline for audio-to-tablature guitar transcription, combining an audio-to-MIDI transcription model, an expressive technique classifier, an audio-conditioned T5 encoder-decoder for string-fret assignment, and an automated tablature generator. In zero-shot evaluation on GuitarSet, EGDB, and two noisy augmented benchmarks, it averages 81.35% audio-to-MIDI F50 (+6.67 over the best baseline), 71.8% string-fret Tab F1 (+8.5), and 54.08% end-to-end Tab F1. The authors state it is the first framework to generate tablature with both fingering and expressive technique annotations directly from guitar audio.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:55 UTC · 6d agoAI research2

A First-Principles Evaluation of Graph-Based Network Intrusion Detection Systems

GIDS-Eval framework reveals evaluation gaps in graph-based network intrusion detection; two crafted edges fully evade three detector-dataset pairs.

Researchers introduce GIDS-Eval, a framework decomposing graph-based network intrusion detection systems into six interchangeable stages to enable controlled comparisons. Surveying nine GIDS and reimplementing five, they find two crafted edges achieve full evasion against three of eight detector-dataset pairs, snapshot windows alone cause a mean 38.3% relative swing in average precision, and none of 18 replayed detector-dataset pairs can alert as events arrive. Their encoder-free GIDS-Lite control ranks first by AP on two of four datasets at up to 575x lower runtime.

arXiv cs.CR·Sep 10, 22:38 UTC · 5d agoResearch1

HyQuant: Hybrid-Precision Quantization for LLM Attention

HyQuant keeps most LLM attention states low-bit while preserving vertical-line tokens and local windows in high precision, maintaining near-lossless accuracy.

HyQuant is a hybrid-precision quantization framework for LLM attention that quantizes most attention states to low bits while keeping accuracy-critical vertical-line tokens and local-window states in full precision, selected via lightweight attention-pattern signals. In the prefill stage it uses a hybrid-precision attention operator, and in the decode stage it applies the same principle to KV-cache compression with fused dequantization and attention computation. Across diverse tasks, models, and datasets it maintains nearly lossless accuracy; code is available on GitHub.

Hugging Face daily papers·Aug 27, 20:00 UTC · 19d agoAI tools & infra1

Ransom Cartel Leader Sentenced to 16 Years in U.S.

Security Affairs·Aug 6, 15:25 UTC · Aug 6, 2026Policy & legal

TempCloze: Can Video-LLMs Identify the Missing Middle?

TempCloze benchmark tests Video-LLMs' temporal reasoning with 1,521 videos, finding temporal alignment is the primary failure mode across 31 models.

TempCloze is a video cloze benchmark in which models must identify the true missing middle clip given the beginning and ending clips, using 1,521 carefully filtered videos from seven sources, mostly long-take and egocentric footage. Distractors are constructed along three dimensions: Semantic, Alignment and Progression, with shared scenes and objects to reduce appearance cues. Evaluation of 10 proprietary and 21 open-source Video-LLMs found Alignment is the primary bottleneck, with models often recognizing plausible semantics and local event progression but struggling with temporal alignment.

Hugging Face daily papers·Aug 31, 20:00 UTC · 15d agoAI research

Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

Researchers introduce the Discovery Certification Protocol, an auditable test framework that verifies whether AI research agents' claimed discoveries are genuine.

The Discovery Certification Protocol (DCP) converts AI research agents' discovery claims into executable recovery and feedback tests organized as gated audits. Controlled audits in SQLite optimization and virtual catalyst control produced zero recoveries in 96 episodes, with an upper bound of 0.0468. A deterministic, LLM-free verifier reproduces audit decisions from frozen evidence, giving AI research a common evidence language for outcomes, alternative routes, and feedback effects.

Hugging Face daily papers·Sep 6, 20:00 UTC · 9d agoAI research

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

IdeaAMBIG benchmark of 660 specification-gap instances shows LLMs localize implementation-critical research gaps poorly, with best model at 9.6% defect recovery.

IdeaAMBIG is a benchmark of 660 evidence-grounded instances evaluating whether research-method specifications provide enough information for faithful implementation: 163 real-world gaps from reproducibility reports and GitHub issues plus 497 controlled synthetic gaps. It tests codification-readiness assessment, defect localization, and clarification action generation across 13 LLMs. The best model achieves only a 9.6% Macro Defect Recovery Rate on real-world instances, though 80.6% clarification success when given the annotated defect, and an oracle study shows gold resolutions raise codification-ready rates from 14% to 98%. Defect localization emerges as the main bottleneck across all evaluated models.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

Fake Software Installers Disable Windows Update and Weaken Microsoft Defender

Fake software-download sites distribute installers that disable Windows Update and weaken Defender, attributed to China-linked cluster Silver Fox.

Microsoft says an active campaign uses counterfeit vendor websites on .com.cn and .hl.cn infrastructure with Chinese-language lures to deliver server-side generated installers that establish scheduled-task persistence, add Defender exclusions, delete shadow copies, and stop services including wuauserv, UsoSvc, uhssvc and WaaSMedicSvc. Victims span healthcare, manufacturing, gaming, technology, logistics, government and education, primarily China-based operations of multinationals and Chinese-speaking users. Microsoft assesses with moderate confidence the activity matches the Silver Fox (Yinhu) cluster, historically tied to Gh0st RAT and ValleyRAT, with C2 over non-standard ports like 5090 and 7088-7090 via domains iualef[.]net and oijfwe[.]net. Kaspersky separately detailed a QN Wallpaper DLL-sideloading chain delivering ValleyRAT.

The Hacker News·Sep 2, 16:41 UTC · 14d agoMalware in the wild

Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

Hugging Face guide fine-tunes a 350M-parameter model with 100 GRPO steps to improve structured output reliability.

A Hugging Face blog post demonstrates fine-tuning a 350M-parameter model using GRPO (Group Relative Policy Optimization) with TRL over 100 training steps. The stated goal is more reliable structured outputs from small language models. No article body was available, so details beyond the title are limited.

Hugging Face Blog·Sep 3, 00:00 UTC · 13d agoAI tools & infra

Understanding the Privacy-Preserving Potential of HTTP/2 Against Webpage Fingerprinting

Researchers show HTTP/2 features can emulate website fingerprinting defenses like FRONT and Tamaraw with tunable privacy-overhead trade-offs.

An arXiv paper demonstrates that application-layer defenses such as HTTPOS, LLaMA, FRONT, ALPaCA, and Tamaraw can be emulated through HTTP/2 features at both the client and server side, including proactive resource suggestion, multiplexing, and flow control. The authors propose a unified evaluation blueprint that calibrates defense parameters per dataset, combines practical attacks with information-theoretic leakage estimators, and measures overheads to map each defense's privacy-overhead trade-offs.

arXiv cs.CR·Sep 4, 13:16 UTC · 12d agoResearch