The correctness illusion in LLM-generated GPU kernels

Every benchmark in the LLM-kernel literature shares one correctness oracle:

torch.allclose(my_kernel(x), reference(x), atol=1e-5, rtol=1e-2)

One shape, one dtype, one seed. KernelBench, TritonBench, GEAK, KernelBand, and STARK all use it. The kernels it blesses are the kernels that ship. And it is structurally blind to a whole family of bugs.

How the illusion works

Consider a softmax kernel that processes its input in tiles of BLOCK columns and forgets to mask the final, partial tile. The bug only fires when the column count isn’t a multiple of BLOCK. Test it on H=256, BLOCK=64 and it’s invisible. Run H=257 in production and every row is wrong — attention shifts by roughly 0.31, degrading a long-context model without ever crashing.

The same pattern repeats across bug classes:

• Accumulator scale — acc = instead of acc +=, masked because the result happens to land within rtol on the chosen shape.
• Missing normalisation — attention without 1/√D, which only saturates softmax differently on some shapes.
• Online-softmax rescale — flash-attention forgetting acc *= α after a max update, wrong only when N > BLOCK_N.

What we measured

We built a 26-op corpus with nine LLM-style buggy kernels and fifteen controls. The one-shape oracle accepted 9/9 of the buggy kernels as correct.

Replacing it with an operator-aware regime — a high-precision fp64 reference, op-schema-aware adversarial inputs, and per-op calibrated tolerances — caught all nine, with zero false positives on the fifteen controls, across five GPU classes (RTX 3060, A10, L40S, A100 SXM4, H100 NVL).

The illusion isn’t that allclose is wrong. It’s that one shape was never enough.

This is the P1 study in the gpuemu research program. See the evidence for P1–P4.