Alignment tax is graded and task-selective in DPO
Length-normalized DPO with SFT retention causes selective capability loss, preserving extraction but harming router tasks.
Stories per day · 30 days
2 RSS
Relatedalignment1dialect1fine-tuning1language-models1llm-judge1llm-training1manaca1over-refusal1portuguese1post-training1preference-data1preference-learning1preference-optimization1reasoning-compression1safety-alignment1sft1training-efficiency1
20 stories, newest first · one card per story
Stories tagged #dpo
LoCo compresses LLM reasoning chains by 41.7-45.4% on math tasks
LoCo trains models to stop overthinking using local preference learning on self-generated traces.
Stochastic DPO graph pruning speeds training about 1.4x
Branch pruning cut training time about 30% versus stock DPO, with slightly lower mean accuracy.
Calibrated accommodation pairs reward dialect matching in post-training
Across 10 open lineages, adjusted DPO still favored British and Australian spelling over AAE and Singlish.
Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge
Manacá-1B-Instruct-2 gained safety from iterated safety finetuning after three failed DPO runs, beating four Portuguese baselines.
LLM-to-Flow Distillation: Teaching Natural Language Goals to Scientific Generators
LLM-to-Flow Distillation uses a frozen LLM and DPO to steer scientific generators toward language goals.
LexReward: A Taxonomy-Driven Reward Framework for Legal Language Models
LexReward defines taxonomy-driven rewards over Style, Element, and Chain dimensions for legal LLMs, feeding DPO and per-dimension reward models for RL.
Smaller Models, Better Rejects: Preference Distillation Scaling
Smaller frozen models produce more useful preference-distillation rejects than a student's own failures on code and math.
Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance
RegLLM measures escalation, grounding, and unsafe actions for regulated agents, but only at smoke scale.
When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safety
Matched tests find DPO controls LLM safety best, while representation probes monitor cheaply and can repair fine-tuning damage.
Improving LLM Collaboration via Multi-Agent Preference Learning
MAPL improves multi-agent LLM collaboration by learning from preferences instead of hard-to-specify joint rewards.
Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models
Researchers build a 12,000-dilemma multilingual dataset and use orthogonalized task vectors to isolate and steer ethical value preferences in LLMs.
How Fyxer built an AI executive assistant people trust
Fyxer details its OpenAI-powered AI executive assistant, orchestrating 30-50 specialized models trained on 500,000+ hours of assistant workflows.
3
Safe Error Correction for Language Models: Frozen-Base Adjustment with Capability Preservation
A 34M logit corrector on frozen Gemma 4 E2B fixes 53.3% of exam errors without capability loss.
StepAudio 3 Music Technical Report
StepAudio 3 Music introduces long-form text-controlled music generation using ABC-notation planning and flow-matching diffusion, ranking near the top music arena.
Domain-Specific Hallucination Detection in Large Language Models
A multi-signal pipeline detects LLM hallucinations, reaching F1 0.915 on HaluEval and cutting Qwen2.5-0.5B hallucination rates from 85.5% to 37.7% via DPO.
RetroThinker: Enabling Retrospective Thinking in Speech LLMs
RetroThinker is a post-training framework letting the Moshi speech LLM self-correct reasoning mid-stream, adding 11% GSM8K accuracy at similar latency.
You Can't Prefer Emotions You Don't Sample: Intensity Undershoot in DPO-Tuned LLMs
Study quantifies DPO-tuned LLMs undershooting requested emotional intensity, tracing the gap to candidate-pool extremity rather than conditioning format.
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
PLC-DPO corrects noisy preference labels during DPO by routing each pair as clean, flip, or tie using calibrated policy-reference margins.
AgenticGen: Reward-Guided Agentic Video Generation for Advertising
AgenticGen applies DPO and GRPO reward-guided reasoning to ad video generation, improving TikTok CTR 2.72%, CVR 2.63%, and Advv 9.61%.