Pterodactyl targets low-cost automated translation quality scoring
1020 Group's WMT26 system uses an LLM judge for segment scores and error-free segment detection.
Stories per day · 30 days
2 RSS
Relateddata-quality1finetuning1gui-agents1instruction-tuning1ios1jailbreak1machine-translation1privacy1privacy-leakage1quality-estimation1red-teaming1reinforcement-learning1ui-navigation1wmt1
14 stories, newest first · one card per story
Stories tagged #llm-as-judge
Online RL framework trains UI agents on live iOS apps
An 8B planning-plus-grounding agent nearly septuples the SFT baseline on Android and ties Gemini-3.1-Pro on real iOS apps.
PrivacyEvalClaw audits privacy leaks in LLM jailbreak replies
The training-free agent gathers evidence and reconciles conflicting judges before scoring indirect or partial personal disclosures.
Response ratio predicts instruction-tuning utility better than common proxies
Under a fixed finetuning budget, answer-dense examples beat common quality proxies on perplexity and LLM-as-judge scores.
Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation
A single colon raised LLM judge Jev's false acceptance of wrong answers from about 1% to 26%.
Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training
Instrument-anchored LLM judges transfer counseling-quality ratings across domains and beat in-domain training.
Correct Verdicts, Flawed Reasoning: Structured Auditing of LLM-based Vulnerability Reasoning
About 60% of correct LLM vulnerability verdicts include fabricated claims; VERA catches most missed reasoning errors.
MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
MetaRubric fixes 'Vacuous Credit' in rubric-based RL using counterfactual prompts and evidence-aware scoring, lifting PubMedQA up to 20.4 points.
AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation
AGO Quality Gate uses judge reliability and probabilistic risk to gate RAG release decisions.
It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
MIST shows irrelevant images change VLM judge labels about as often as aligned images, without improving agreement.
PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?
PlaylistEval tests video-language judges on 100-hour playlists, where frontier models reach only 75.4% accuracy.
Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification
A medical RAG study finds retrieve-then-verify factuality checks fail for structural reasons, not weak models.
Optimal Sequential Annotations for Off-Policy Evaluation
Paper optimizes sequential annotation budgets for doubly robust off-policy evaluation of complex outcomes.
VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering
VakQA debuts a 2,001-pair Telugu spoken factoid QA benchmark, validating LLM-judge evaluation and benchmarking speech-input model reliability.