Search: “model-evaluation”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalization

ModerationBench shows foundation models can nearly triple Bluesky's moderation F1 (0.60 vs 0.22), with instruction- and example-driven guidance performing comparably.

Researchers built ModerationBench, a new benchmark of 4,000 manually annotated in-the-wild posts from Bluesky, to test whether foundation models can reliably operationalize content moderation policies. They systematically compare instruction-driven guidance (reasoning from policy precepts) with example-driven guidance (generalizing from precedents) for Vision-Language Models. Both paradigms achieve comparable peak effectiveness, and foundation models nearly triple the F1 of Bluesky's deployed moderation system on Random Posts (0.60 vs 0.22).

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:30 UTC · 7d agoAI research1

28

Causal Foundation Models

A paper introduces causal foundation models (CFMs): pretrained networks that estimate treatment effects on new datasets via in-context learning without fine-tuning.

Causal foundation models (CFMs) apply the foundation-model paradigm to causal inference, replacing bespoke per-problem estimator pipelines with networks pretrained once at scale. CFMs estimate causal quantities such as the average treatment effect on entirely new datasets through in-context learning, without model updates. The work serves as a practical introduction to the emerging area, covering background in causal inference and machine learning and including example code and Jupyter notebooks.

Hugging Face daily papers·Sep 1, 20:00 UTC · 14d agoAI research

40

Thought without systematicity? Evaluating reasoning models on rule induction tasks

Study finds reasoning models often fail on structurally equivalent variants of tasks they solve, suggesting their reasoning lacks systematicity.

The paper extends rule induction tasks from cognitive science using task isomorphisms such as recombination and substitution to test systematicity in reasoning models. Despite solving tasks correctly, models frequently fail on structurally equivalent variants of the same task. The authors conclude many model behaviors lack systematicity, making it difficult to establish cognitive abilities beyond the specific evaluation contexts.

Hugging Face daily papers·Sep 11, 20:00 UTC · 4d agoAI research

35

Competence-Gated Pooling of Language Models and Priors for Event Forecasting

Paper proposes a competence gate pooling language model forecasts with external priors, improving Brier score from 0.0771 to 0.0732 across 2,357 binary questions.

The paper defines a language model's relative competence as its marginal value beyond an available external forecast, and derives conditions under Brier loss where model disagreement improves that forecast. A competence gate estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, it improves the external baseline from 0.0771 to 0.0732 Brier and beats global forecast combinations, though it defers to the market on ForecastBench. Across four Qwen models, verbal confidence failed to identify when the model outperformed the external forecast, while outcome-estimated competence supported better abstention.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

38

Generative Marketing Mix Modeling: A Causal Inference Framework Linking GEO and GEM to Business Impact

Paper proposes Generative Marketing Mix Modeling to causally estimate Generative Engine Optimization and Marketing effects on business outcomes.

The authors develop GMMM, a causal inference framework for measuring how often users see and notice a firm's name in generated answers, which standard marketing data ignore. For GEO it combines repeated generated answers with question counts, shares of generative-system usage and notice probabilities; for GEM it uses sponsored placement records with notice probabilities. The framework compares expected business responses under alternative treatment sequences, establishes identification conditions, and is evaluated on simulated product-recommendation answers in English and Japanese.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:57 UTC · 5d agoAI research

20

ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation

ActReview post-trains Qwen3-8B-Base on OpenReview rebuttals to generate actionable peer-review feedback with grounded revision suggestions, benchmarked on 1,000 curated instances.

The paper defines Actionable Peer-review Generation as diagnostic claim generation plus revision suggestion generation and introduces ActReview, a rebuttal-guided post-training framework. From OpenReview review-rebuttal threads the authors build ActReview-40K, aligning reviewer weaknesses with author responses grounded in localized paper evidence, and post-train Qwen3-8B-Base with multi-task SFT followed by GRPO using weakness-specific rubric rewards. They also release ActReview-Bench, a human-curated 1,000-instance benchmark, on which ActReview outperforms prior specialized review-generation models on actionability and grounding while remaining competitive with strong prompt-based LLMs. Human evaluation confirms improved revision usefulness but identifies a remaining gap in technical accuracy.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

30

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

DataFlex-RL benchmark of 13 RLVR data policies on Qwen2.5-7B finds none reproducibly beats uniform sampling under matched GRPO training.

DataFlex-RL is an evaluation platform comparing rollout-selection, reweighting, and mixture data policies for RLVR under a common GRPO recipe. Across 13 configurations and 12 matched seeds with Qwen2.5-7B-Base on 12 math, logic, and science benchmarks, uniform GRPO improved domain-balanced accuracy by 7.76 points, but no alternative policy achieved a statistically significant improvement. A corrected 12-seed Llama-3.1-8B-Base extension found no consistent winner, and math-heavy evaluation summaries were negatively correlated (-0.33) with domain-balanced summaries.

Hugging Face daily papers·Sep 4, 20:00 UTC · 11d agoAI research

35

AI for Games in the Foundation Model Era

Survey organizes foundation-model AI for games into six roles and analyzes which capabilities transfer across playing, design, building, runtime adaptation, and testing.

A survey maps foundation-model and learned world-model research across the game lifecycle into six roles: playing/acting, modeling players and games, designing games, building/maintaining games, runtime generation/adaptation, and testing/evaluation. The authors identify cross-role connections such as trajectories training world models and design specifications driving executable implementations. Control schemes, rules, engine interfaces, state representations, and player contexts often remain setting-specific, so downstream claims require validation in the target setting. Evaluation is most standardized for bounded game playing, while persistent state, repeated revision, validated player modeling, and automated testing remain less established.

Hugging Face daily papers·Sep 14, 20:00 UTC · 1d agoAI research

25

OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining

OpenWAM releases an open modular stack for world-action model pretraining, plus OpenWAM-alpha trained on about 6,400 hours of egocentric and robot data.

OpenWAM is an open research stack that factorizes World-Action Model pretraining into composable infrastructure, study, and model components with unified training, inference, and evaluation. Controlled experiments distill three principles on knowledge inheritance, world-action synergy, and out-of-domain generalization gains from embodied co-training. The resulting OpenWAM-alpha, pretrained on roughly 6,400 hours of egocentric human and robot data, achieves top-tier results across eight simulation benchmarks and real-robot tests spanning single-arm, bimanual, and dexterous embodiments. The full stack, including infrastructure, evaluation protocols, pretrained models, and data recipes, is released openly.

Hugging Face daily papers·Sep 6, 20:00 UTC · 9d agoAI research

44

Bag of Tricks or Bag of Myths? Reducing Modeling Complexity with Task Knowledge in Explainable Suicide Risk Assessment

Researchers audit 31 NLP techniques for clinician-annotated suicide risk prediction, finding only 5 of 31 comparisons yield reliable gains.

A study of 1,635 clinician-annotated social media posts ran roughly 300 controlled experiments across 7 methodological families, auditing techniques such as model scaling, synthetic data, ensembling, and threshold tuning under severe class imbalance. The proposed system reformulates risk factor prediction as entailment between posts and codebook definitions, using architecturally diverse ensembles with class-balanced training and deployment-consistent calibration. It scores 0.8203 for risk, 0.7953 for evidence, and 0.7045 macro-F1 for factors, ranking third among 53 teams.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:14 UTC · 9d agoAI research

14

Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence

New framework tests whether LLM-cited explanation factors are necessary or sufficient, finding weak correlation across Claude, GPT, and Gemini models.

An arXiv paper introduces black-box intervention tests measuring whether factors LLMs cite in their explanations are necessary or sufficient for their outputs in agent oversight workflows. Across eight models from the Claude, GPT, and Gemini families, Spearman correlations between cited rankings and measured influence ranged from 0.349-0.354 (advisor recommendation) to 0.431-0.580 (prompt monitoring). Uncited factors scored above the lowest cited factor in up to 57.6% of advisor responses, showing cited top-three factors do not reliably identify the most influential inputs.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:37 UTC · 11d agoAI safety & security1

44

Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation

OPRD distillation enables weak-to-strong generalization by amplifying verifier-supported policy updates, outperforming existing RL and distillation methods with fewer student updates.

On-Policy Reverse Distillation (OPRD) evaluates a weak teacher's policy shift relative to its reference policy on student rollouts and amplifies the verifier-supported component of the student's policy gradient. This rescaling preserves the stationary points of policy optimization while letting the student learn beyond the teacher's capacity ceiling. In successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches, and response-style analysis shows students remain closer to verifier-RL-trained models than to their weak teachers.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

33

ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation

ActReview post-trains Qwen3-8B-Base on 40K rebuttal-derived instances with rubric rewards to generate actionable, grounded peer-review feedback, plus a 1,000-instance benchmark.

The framework builds ActReview-40K from real OpenReview review-rebuttal threads, aligning reviewer weaknesses with author responses and grounding feedback in localized paper evidence. Qwen3-8B-Base is post-trained with multi-task supervised fine-tuning followed by GRPO using candidate-aware, weakness-specific rubric rewards. Experiments show improved actionability and grounding over prior specialized review-generation models, supported by ActReview-Bench, a human-curated 1,000-instance evaluation set. Human evaluation confirms better revision usefulness while noting a remaining gap in technical accuracy.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:30 UTC · 7d agoAI research

31

From Reweighting to Rewriting: Unlocking the Intervention Effects of Influential Samples in Training Data Attribution

Study shows rewriting responses of influence-selected training examples shifts LLM behavior more strongly than reweighting the same samples.

The paper examines training data attribution, arguing that influence functions identify high-leverage examples whose value goes unrealized under conventional weight-based reweighting interventions. It introduces influence-guided response rewriting, which replaces the responses of influence-selected examples with behavior-aligned or behavior-opposed supervision while keeping instructions fixed, tested across four open-weight LLMs using epistemic abstention as the primary testbed. Rewriting produces stronger, more persistent, and bidirectional behavioral shifts, including on safety refusal, while reweighting the same examples yields weak, inconsistent effects. The results motivate intervention-aware evaluation of TDA methods.

Hugging Face daily papers·Sep 1, 20:00 UTC · 14d agoAI research

30

CausalArena: Benchmarking Causal Discovery in the Foundation Model Era

Researchers introduce CausalArena, a unified benchmark revealing that causal discovery rankings shift substantially across structural causal model families and protocols.

The paper presents CausalArena, a unified and evolvable benchmark for causal discovery combining synthetic structural causal models, semantically grounded operational SCMs, formula-grounded scientific SCMs, and public real-world datasets. Experiments across classical, neural, and pretrained causal discovery foundation models show large ranking shifts between benchmark regimes. The authors identify pretraining-evaluation overlap and benchmark diversity as central evaluation challenges.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:53 UTC · 5d agoAI research

30

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

GE-Act 2.0 is a from-scratch pretrained world-action model for robotic manipulation, with success rising from 17.1% to 44.1% as co-training data scales to 30,000 hours.

Genie Envisioner Act 2.0 (GE-Act 2.0) is a world-action model whose generative and action components are all initialized from scratch on manipulation data, combining a control-oriented autoencoder (CoAE), single-step visual planner (SVP), and inverse dynamics model (IDM) trained jointly via knowledge-aligned selective optimization (KASO). Scaling co-training data from 300 to 30,000 hours raises zero-shot success from 17.1% to 44.1% on G1-OP and 13.4% to 31.1% on G2-90D, despite the latter comprising under 2% of data, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage correlates with zero-shot OOD success (Pearson r=0.80).

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research

30

When LLM judges agree, should we believe them?

Amazon ICML paper uses Ising models to correct correlated LLM-judge votes, beating accuracy-weighted panels by 9-14%.

Amazon Science describes an ICML paper, "Dependence-aware label aggregation for LLM-as-a-judge via Ising models," addressing how correlated judge outputs inflate majority-vote confidence. The unsupervised method models pairwise dependence between judges, learning both reliability and similarity without human reference labels. Tested on relevance, toxicity, and summarization tasks with 10 judge models at temperature zero, it outperformed accuracy-weighted voting by 9% to 14%.

Hacker News · AI·Sep 14, 16:29 UTC · 2d agoAI researchHN 41↑ · 23 comments

40

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

Clinician-calibrated K-Bench evaluates 125 LLM configurations on 200 high-risk mental health vignettes, exposing wide variation in suicide and violence risk handling.

K-Bench is a clinician-calibrated, protected benchmark evaluating 125 model configurations from 33 base models across 14 providers on 200 multi-turn vignettes covering suicide, self-harm, domestic violence, substance misuse and no-risk presentations. A frozen GPT-4o judge achieved 94.2% exact agreement with clinician consensus across 6,751 eligible comparisons from 151 clinician-rated transcripts. Leading models combined supportive conversation with combined-risk scores above 95, while risk exploration varied substantially among weaker configurations; therapeutic prompting helped weaker models and elevated reasoning produced no average improvement. A continuously updated public leaderboard is hosted at k-bench.ai with protected test materials.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 16:49 UTC · 2d agoAI safety & security

46

Scaling Automatic Research Agents via World Models

WMRL replaces environment execution with a world model in RL, accelerating research-agent post-training 3-4x and letting 4B/9B agents beat 48B/120B open-weight agents.

The paper identifies that environment execution dominates RL training cost for automatic research agents because each execution occupies an exclusive sandbox while generation batches efficiently. World Model RL (WMRL) substitutes a learned world model for execution, with Online Debiasing and Inverse-Variance Denoising to handle reward bias and noise, and the paper proves both improve convergence guarantees. WMRL accelerates training 3-4x across tasks and outperforms standard RL baselines; post-trained 4B and 9B agents beat 48B and 120B open-weight agents on held-out benchmarks. WMRL also transfers to post-training embodied VLA policies.

Hugging Face daily papers·Aug 28, 20:00 UTC · 18d agoAI research1

45

ActionSplice: In-Flight Action Editing for Interactive World Models

ActionSplice enables in-flight action editing in chunk-autoregressive video world models via a lightweight corrector, avoiding rollback or waiting for the next chunk.

ActionSplice is an inference framework that formulates in-flight action editing for chunk-autoregressive video world models as Counterfactual State Transport (CST), where a lightweight corrector transports the interrupted backbone-native representation toward the matched state induced by the revised action. The world model and sampler remain frozen, and sampling resumes without replaying completed evaluations. Across minWM-Wan Action2V and HY-WM1.5, the retargeting variant CST-R reduces rollback-relative LPIPS by 61.5% and 75.9% versus direct condition swapping, while the temporal-splicing variant CST-T reduces suffix LPIPS by 56.1% and 77.5% with 2.73x and 1.69x pixel-ready speedups over waiting.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

22

Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models

Audits of 10 classifiers on BRFSS show target leakage, not model class, drives the reported 0.89 AUROC in survey-based cardiovascular screening.

The study benchmarks ten model classes, including glass-box and tabular foundation models, for prevalent myocardial infarction on 442,067 respondents of the 2022 BRFSS across five feature tiers of decreasing leakage risk. Removing two post-diagnostic features costs every model 0.049-0.051 AUROC and collapses performance into a 0.0045-wide band, and the explainable boosting machine matches all alternatives within 0.005 while scoring roughly 104x faster than the strongest foundation model. Frozen models transport within 0.002 AUROC to 2023 data; the authors conclude evaluation practice and feature sets, not model capacity, are the binding constraint.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:29 UTC · 5d agoAI research1

28

Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps

Hugging Face guide fine-tunes a 350M-parameter model with 100 GRPO steps to improve structured output reliability.

A Hugging Face blog post demonstrates fine-tuning a 350M-parameter model using GRPO (Group Relative Policy Optimization) with TRL over 100 training steps. The stated goal is more reliable structured outputs from small language models. No article body was available, so details beyond the title are limited.

Hugging Face Blog·Sep 3, 00:00 UTC · 13d agoAI tools & infra

22

Meta FAIR Introduces AI Research Preference Models (RPMs): Ranking ML Experiments Before Spending GPU Hours

Meta FAIR, Oxford and UCL introduce Research Preference Models that rank unexecuted ML experiments, lifting AIRS-Bench scores from 0.684 to 0.729 and cutting compute ~1.6×.

Researchers from Meta FAIR, Oxford, and UCL introduce Research Preference Models (RPMs), which use frozen pretrained LLMs (Qwen3.6-27B backbone, no fine-tuning) to rank unexecuted experiment candidates and execute only the winner of a pairwise knockout tournament. Two variants shipped: an inference-only LLM-as-a-judge and an agentic variant that runs small pilot experiments in an H200 sandbox. On AIRS-Bench (20 tasks, 24 hours on one H200, 10 seeds), scores rise from 0.684 (random) to 0.711 and 0.729 versus a 0.748 validation oracle, and both variants reach the baseline's 24-hour score in roughly 15 hours. The team reports new SOTA on WinoGrande (94.1% with Agentic RPM) and SVAMP (95.7% with inference-only).

MarkTechPost·Sep 6, 20:25 UTC · 9d agoAI research

45

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability

A controlled study finds agent memory portability varies sharply: fixed-schema knowledge graphs survive model swaps while compressed notes degrade.

The study compares preserving an agent's history as raw long context, RAG chunks, compressed natural-language notes, or fixed-schema knowledge graphs across model upgrades, using 48 synthetic histories and two open-weight sub-10B-parameter models. Fixed-schema KG accuracy changed by only +0.0004 ± 0.0020 after a writer swap, while compressed NOTES shifted asymmetrically by +9.91 or -13.28 percentage points depending on migration direction. Mixed 50/50 embedding migrations captured only 4.96 of an 11.90-point RAG re-embedding gain; 80% of the NOTES deficit came from information lost at construction, and 81% of the RAG deficit from retrieval failures. Store-only repair of NOTES failed to reach 90% recovery in all 48 cases, while retaining raw histories enabled recovery in 34 of 48 for one direction.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 16:44 UTC · 12d agoAI research1

32

BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure

BenchShield uses lifecycle-model-backed instrumentation to detect reward hacking in LLM-agent benchmarks, lifting full-chain recall to 77-100% at up to 65% lower cost.

The framework grounds reward-hacking detection in a finite lifecycle model of an evaluation's reward-relevant events, combining a static phase-aware taint analysis with runtime infrastructure-side evidence attribution. Evaluation used a human-labeled corpus of 456 adjudicated trajectories drawn from more than 31,000 public agent runs across three benchmarks. BenchShield improves full-chain recall from 23-94% to 77-100% and same-vector coverage from 16-56% to 43-78%, cuts per-task cost by up to 65%, and achieves 96% accuracy detecting reward hacking at runtime.

arXiv cs.CR·Sep 10, 03:10 UTC · 6d agoAI safety & security1

42

Evaluating Time-Series Foundation Models and Multimodal Dietary Context for CGM Forecasting

Study finds zero-shot time-series foundation models underperform on CGM forecasting; fine-tuned Chronos-Bolt cuts RMSE up to 18.4% and dietary context adds signal.

The paper evaluates time-series foundation models for continuous glucose monitoring forecasting across eight public datasets covering Type 1 diabetes, Type 2 diabetes, and non-diabetes populations. Under a unified protocol, zero-shot foundation models did not consistently outperform baselines like Elastic Net and PatchTST, but lightweight fine-tuning did, with fine-tuned Chronos-Bolt reducing RMSE by 6.5%-18.4% in the T1D cohort and 8.6%-18.2% in the non-diabetes/T2D cohort. A residual-based fusion framework adding dietary context from CGMacros reduced overall RMSE by about 3% and postprandial RMSE by about 15% versus CGM-only baselines.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:43 UTC · 5d agoAI research

28

The Frontier AEO Tracker: What Astra Chooses (and every other frontier model, and what you can do about it)

Latent Space launches an AEO tracker scoring 7 frontier models' product recommendations across 161 categories, revealing generational bias flips.

Latent Space built a tracker measuring Answer Engine Optimization by running 6 prompt variations across 7 frontier models with search enabled over 161 product categories, scoring first choices, alternatives, mentions, and anti-recommendations. It found 28 categories with a universally dominant primary choice and observed soft biases, such as models favoring their own lab's coding agents. Analysis of Anthropic's Sol→Astra and Opus→Fable generations showed newer models consulting fewer sources and being less likely to change answers when questions are paraphrased.

Latent Space·Sep 7, 21:32 UTC · 8d agoAI research

35

ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

ReactHuman benchmark tests whether multimodal LLMs react safely to sudden household hazards; seven evaluated models mishandle roughly one hazard in three.

ReactHuman is the first physics-grounded benchmark for human-like reactive decision-making, placing a multimodal LLM as the brain of a simulated humanoid facing 17 event families of sudden household hazards across over 1,000 bit-for-bit reproducible scenes with annotation-free ground truth from 240 Hz rigid-body simulation, including adversarial objects whose appearance contradicts their physics. A five-metric suite scores each reaction along reasonable, safe, and physically grounded axes, and every committed plan is physically executed. Seven representative MLLMs mishandle roughly one hazard in three, act from fixed dispositions rather than the observed scene, trust appearance over motion, and miss interception points at meter scale; none of these failures shrink with model scale.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

42

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

A 400-task BigCodeBench evaluation shows frontier LLMs widely over-edit code; a preservation instruction cuts excess edits and raises Pass@1 by 2.3 points.

Researchers built an evaluation framework from 400 BigCodeBench problems with injected AST-level corruptions, each with a known minimal patch, to measure over-editing in LLM code repair. Even strong models like GPT-5.5 produce unnecessarily large edits despite high Pass@1. Adding a preservation instruction reduced average excess Levenshtein distance from 0.195 to 0.131, cut added cognitive complexity by 26.6%, and raised Pass@1 by 2.3 points. Reinforcement learning post-training gave the best out-of-domain edit-fidelity trade-off, while supervised fine-tuning overfit to seen corruption patterns.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research1

30

Reference-Based Bias Detection in LLMs via Relative Representations of Hidden States

Researchers propose auditing LLM bias via relative hidden-state representations, detecting bias increases with 3-50x less compute than output-level benchmarks.

The paper introduces a reference-based bias auditing method that compares hidden-state representations across model variants, such as before and after fine-tuning, by encoding sentences relative to a fixed anchor set. The resulting Representational Bias Shift (Delta-B) correlates with output-level bias change in 15 of 18 tested settings, reaching |r| = 0.84 under full fine-tuning across WildGuardMix, DecodingTrust, and ToxiGen benchmarks. Thresholding Delta-B detects checkpoints whose bias increased with ROC AUC between 0.65 and 0.99 and beats a SEAT-based baseline, while auditing a model in about three minutes with 3-50x less compute.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

42