What silent GPU-kernel bugs actually cost

The dangerous failures in GPU code aren’t the ones that crash. They’re the ones that don’t.

The cost has three parts

GPU-hours billed for broken work. A miscompiled matmul propagates through every forward pass. Training burns tens of thousands of GPU-hours per epoch on H100/A100 — and a silently-wrong kernel means every one of those hours computes the wrong thing. The compute is still billed.

Quality regressions with no stack trace. A broken flash-attention degrades long-context quality without crashing. An unmasked reduction taints metrics no one watches. There’s no error to grep for — just a model that’s quietly worse, found weeks later if at all.

Months of green CI. Because the correctness gate is a one-shape allclose, the bug never turns CI red. Teams ship with confidence on top of a suite that structurally cannot see the failure.

It’s measurable, and it’s a recognised problem

PyTorch’s own triage data shows 19.2% of high-priority torch.compile issues are silent wrong-output — the second-largest category after crashes. Silent Data Corruption in numerical code is now a recognised academic sub-field for exactly this reason.

Making it loud

The fix isn’t more compute — it’s a gate that can see the failure. gpuemu turns silent wrong-output into a red CI check with a replayable seed: an fp64 reference oracle, op-schema-aware adversarial inputs, and calibrated per-op tolerances, all running without a GPU.

See what it costs and the evidence for the full picture.