Neumann says GenDB-V benchmark gains come from query overfitting
Original title: Next-Level Benchmarking Overfitting – Thomas Neumann
Reported 22× to 61× speedups over Umbra and DuckDB rely on precomputed structures and hard-coded TPC-H layout.
Thomas Neumann argues that AI agents optimizing for TPC-H overfit the benchmark, using GenDB-V as the example. The paper reports 22× to 61× speedups over Umbra and DuckDB in a formally verified system. Neumann says agents precompute per-query structures and hard-code dbgen row layout, falling back to a slow path if those assumptions fail, so the gains would not hold in a production system.
Unconfirmed: The bandwidth bound and infeasibility argument are Neumann's analysis, not results reported by the GenDB-V paper.
- At SF10, Umbra finishes 22 TPC-H queries in about 540ms on a 9950X3D.
- Neumann estimates memory bandwidth alone takes about 280ms for the touched data.
- A plain-array mode still reports roughly a 3× improvement.
- At SF100, Q2, Q17, and Q4 run near constant time in the tuned design.
From the original article
3 min read at databasearchitects.blogspot.comblog post I remarked that database benchmarking is in a crisis because AI agents start to overfit on individual benchmarks with questionable techniques, which drives execution times towards zero. This
Hacker News · securityBy artrax · databasearchitects.blogspot.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.