HALO evaluates proactive agents with human annotations
Original title: HALO: Human-Grounded Granular Evaluation of Proactive Agents
HALO-8K dataset tests 10 models across 400 scenarios, revealing over-proactivity in 61.20% of rejections.
Current evaluations of proactive agents lack granular human diagnosis of evidence use and intervention timing. HALO-Arena spans 4 need types in 10 domains and 50 environments. Expert annotations on HALO-8K responses highlight timing and intensity issues as primary failure modes.
- HALO-8K contains 8,000 responses
- Proprietary models reach 72.88% acceptability
- Open-source models reach 56.63% acceptability
- Over-proactivity dominates rejections
- Covers Dependency, Blockage, Risk, Opportunity needs
- AI models
- HALO
From the original article
1 min read at huggingface.co
Proactive agents must decide whether, when, and how far to assist without an explicit request. Existing evaluations often rely on automated scores or binary verdicts, offering limited human diagnosis of evidence use and intervention quality. To address these gaps, we introduce HALO, Human-Grounded GrAnular EvaLuation of PrOactive Agents.
By Wenqi Pei, Xinyu Wang, Jinrui Liu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.