WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data
WearableQA benchmark tests LLM health reasoning over longitudinal wearable data; the best of 14 evaluated LLMs reaches 72.9% accuracy.
WearableQA comprises 4,084 ten-option multiple-choice questions built from wearable time series, blood biomarkers, and demographics of 200 real users with up to 500 days of daily measurements. It defines 16 question types along two axes: data versus health reasoning, and single- versus cross-signal reasoning. Evaluation of 14 proprietary and open-source LLMs shows performance from 19.6% to 72.9% against a 10% chance baseline, with most models below 60%.
- Benchmark uses real longitudinal wearable data preserving device noise and variability
- Dual-grounding framework combines literature and population-validated physiological patterns
- Most evaluated LLMs score below 60%, showing the benchmark is far from solved
From the original article
1 min read at arxiv.orgRecent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whether AI systems can reason over a real user's longitudinal wearable record.
By Ji Soo Lee · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.