Neumann says GenDB-V benchmark gains come from query overfitting
Reported 22× to 61× speedups over Umbra and DuckDB rely on precomputed structures and hard-coded TPC-H layout.
Stories per day · 30 days
1 RSS
Relatedai-agents1databases1energy-efficiency1evaluation1gendb-v1gqa1kv-cache1llm-inference1red-teaming1tpc-h1web-agents1
13 stories, newest first · one card per story
Stories tagged #benchmarking
PKPS enables benchmark prediction with sparse responses
PKPS represents models to predict benchmarks from sparse unpaired data.
SIMPLE: a four-parameter law predicts LLM inference energy cost from architecture
Across 13 dense LLMs on H100 GPUs the law reaches R2 of 0.969 and 11.97% MAPE, and shows agentic requests cost 13x more energy than non-agentic ones.
ANGLER adaptive attacker raises web agent success rate to 20.36%
ANGLER near-miss-guided framework boosts end-to-end attack success on web agents to 20.36% while lowering benign task success from 61.9% to 44.4%.
Cloud Range lets SOCs benchmark AI agents against human defenders
Cloud Range launches AI Validation Range and PROVE-based AI Readiness Framework to red team AI agents and benchmark them against human SOC defenders.
1
The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
Researchers introduce Taste-Bench, a benchmark measuring LLM agents' ability to make good long-horizon decisions at critical decision forks.
Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Study shows LLM test-time scaling cost depends on generation schedule: eight serial A100 calls use ~4.7x more GPU energy than one batched call.
Implementation of Machine Learning Workflows with NVIDIA cuML, RAPIDS, GPU Benchmarking, Explainability, Clustering, and Model Inference
Hands-on tutorial implements NVIDIA cuML and RAPIDS to GPU-accelerate scikit-learn-style ML workflows with benchmarking, clustering, and inference.
2
Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Study shows visually grounded token embeddings in a small masked LM persist through training and improve object-property knowledge, but escape standard BabyLM benchmarks.
Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses
Quesma benchmarks Qwen3.8 27B quantizations: 4-bit Q4_K_M matches BF16 on key benchmarks while 1-bit collapses to random chance.
10
ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing
ACEA is a pluggable arena scoring LLM red-team attackers and blue-team defenses head-to-head with an LLM judge and verifiable leakage ground truth.
PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving
PlannerForge unifies scenario-based testing of autonomous driving motion planners in one LLM-agent framework, outperforming prior baselines.
Piloting the world's first double-blind AI evaluations
Google DeepMind is piloting the world's first double-blind AI evaluations, a new methodology intended to improve evaluation integrity and reduce bias.