gpuemu vs KernelBench, TritonBench, GEAK, and the LLM-kernel benchmarks
KernelBench, TritonBench, GEAK, KernelBand, and STARK are leaderboards for LLM-generated kernels — and they use the same one-shape allclose oracle inside. gpuemu is the user-facing correctness tool that oracle should have been.
| Capability | gpuemu | KernelBench / TritonBench / GEAK |
|---|---|---|
| User-facing correctness tool | Yes | No |
| fp64 reference oracle (vs one-shape allclose) | Yes | No |
| Adversarial / boundary inputs | Yes | No |
| Per-op calibrated tolerances | Yes | No |
| Runs in your CI on your kernels | Yes | No |
KernelBench, TritonBench, GEAK, KernelBand, and STARK are leaderboards: they measure how well an LLM or agent can generate GPU kernels. They’re useful for that.
But two things make them the wrong correctness tool for your kernels. First, they’re not
user-facing — you don’t point them at your repo and gate your CI. Second, and more important,
the oracle they score against is the same one-shape allclose that accepted 9/9 buggy
kernels in our corpus. A kernel can top a leaderboard and still be silently wrong.
Notably, STARK’s own description references an internal “five-stage correctness harness” — which mirrors what gpuemu does in the open. The need is real; gpuemu is the tool you can actually run.
Frequently Asked Questions
Can I use a benchmark like KernelBench as my correctness gate?
No — these are leaderboards for comparing LLM kernel-generation systems, not tools you run on your own kernels in CI. And the oracle they score against is the same one-shape allclose that missed 9/9 bugs in our corpus.