Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization
Mechanical checks plus LLM review cut errors when verifying Lean formalizations of NIST math functions.
Stories per day · 30 days
2 RSS
Relatedautoformalization1dlmf1dpo1formal-verification1lean1manaca1mathematical-functions1over-refusal1portuguese1safety-alignment1sft1
9 stories, newest first · one card per story
Stories tagged #llm-judge
Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge
Manacá-1B-Instruct-2 gained safety from iterated safety finetuning after three failed DPO runs, beating four Portuguese baselines.
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
A position-masking self-distillation method trains LLM judges from language feedback and beats outcome-only RL on subjective tasks.
Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures
RLCDAlignBench benchmarks Jev, a calibrated-decision model detecting ten AI alignment failure types zero-shot at 0.886 median AUROC, 63x cheaper than LLM judges.
OSWorld-Pro: Process-based Evaluation for Computer Use Agents
OSWorld-Pro adds process-level subgoals so computer-use agent failures can be diagnosed beyond final outcomes.
OSWorld-Pro: Process-based Evaluation for Computer Use Agents
OSWorld-Pro scores computer-use agents on over 300 tasks; Claude Opus 5 reaches 75.7%.
Forgeable Confirmation in Automated Computer Security Testing: Deterministic Rules versus AI Judges
Checks that trust attacker-controlled data can forge success verdicts in automated AI security testing.
Multi-Dimensional Prosody Judgment For Live Streaming Speech Synthesis
Researchers introduce Live-ProsodyJudge and D-LPJ, Gemini-distilled Qwen3-Omni judges that decouple multi-dimensional prosody scores for live streaming TTS evaluation.
When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation
Training LLM reviewers on model-generated reviews compresses rating distributions and diversity, a failure the authors name scientific-judgment collapse and mitigate with TrustReviewer.