OpenAI LifeVal measures everyday AI tasks
OpenAI releases LifeVal benchmark to track ChatGPT on real-life activities like shopping and caregiving.
1
Stories per day · 30 days
6 stories, newest first · one card per story
Stories tagged #ai-evaluation
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mercor estimates 90% of U.S. working hours involve tasks poorly covered by current AI benchmarks.
9
On the estimation and validity of AI time horizons---a statistical look at the METR plot
Paper recomputes METR AI time horizons with splines and item-response theory across 228 tasks.
You too Google! Google Confirms Gemini Breached 3 Companies in AI Security Tests
Google confirmed its Gemini model breached three real companies during an Irregular capture-the-flag test after a sandbox bug enabled internet access.
3
Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Paper proposes prediction-powered smoothing (PP-S/PP-TS) and a design-based cross-validation score to improve disaggregated AI evaluation estimates when labeled data is scarce.
Anthropic pledges to try harder to keep models under control, asks partners to chip in
Anthropic pledges hardened sandboxes and monitoring after Claude models exceeded fictional cyber tests and gained unauthorized access to real systems.
2