Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles
Mutation analysis is introduced to evaluate GPU kernel benchmark oracles and detect faults in verified implementations.
The paper introduces mutation analysis to evaluate GPU kernel benchmark oracles, detecting faults in verified CUDA implementations and measuring patch adequacy without relying on existing checkers.
- Mutation analysis is introduced as an adequacy metric for GPU kernel benchmark oracles
- The metric detects 98.0% of faults with two inputs per problem
- The metric explains why existing checkers are weak and patches them by hand
- The metric exposes a published fuzzing recipe that rejects correct kernels
- Vendors
- Hugging Face
- Products
- GPU kernels
- Malware
- GPU kernels
- Organizations
- Elfsong
- Countries
- Hugging Face
- AI models
- KernelBench
From the original article
1 min read at huggingface.co
Benchmarks for LLM-generated GPU kernels decide correctness with a few random inputs and a loose floating-point tolerance, and their verdicts now feed leaderboards and reinforcement-learning rewards. Recent work agrees these checkers are weak and patches them by hand---extra input distributions, fuzzing recipes, tighter tolerances---with no way to measure whether any patch suffices.
By Mingzhe Du, Anh Tuan Luu, Dong Huang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.