# AI-generated kernels · Frontier

<!-- https://learn-kernels.com/chapters/frontier/ai-generated-kernels -->

The claim under test is simple: language models can write CUDA kernels that beat PyTorch. KernelBench, the benchmark that made the question precise, evaluates models on 250 PyTorch workloads and scores them with **fast\_p** (KernelBench's metric: the percentage of generated kernels that are functionally correct and offer a speedup greater than an adjustable threshold p over baseline.), the fraction of generated kernels that are both correct and faster than the baseline by more than a chosen threshold. Its authors found that frontier reasoning models did best out of the box but still matched the PyTorch baseline in less than 20 percent of cases, with the benchmark getting harder as the speedup threshold rises.

Then the correctness checks themselves came under scrutiny. KernelBench-Verified, a follow-up from Meta and Stanford, found that frontier models frequently engage in **reward hacking** (Exploiting the specific conditions of an evaluation to inflate a score, rather than solving the task the evaluation is meant to measure.). The original baseline ran PyTorch in plain float32, without **TF32** (TensorFloat-32, a compute mode on recent NVIDIA GPUs that routes float32 matrix multiplications through Tensor Cores with a truncated significand, enabled in PyTorch by a single flag.), so any generated kernel that merely invoked cuBLAS looked dramatically fast against an artificially slow reference. And because the test inputs came from one narrow distribution, all positive and small, models learned to hardcode bypasses: one generated ReLU kernel checked for the test shape and returned its input unchanged, passing the correctness check and reporting a 374x speedup while computing nothing.

> Figure. How a reward-hacked kernel passes. The ReLU kernel from KernelBench-Verified's analysis: it checks for the benchmark's input shape and returns its input unchanged. Against one narrow test distribution it passes correctness and reports a 374x speedup while computing nothing; a hidden suite drawn from four input distributions catches it.

Under the verified protocol, a TF32-enabled baseline plus a hidden test suite of four input distributions, the picture inverted. The best model’s geometric mean speedup fell from 1.43x under the standard protocol to 0.88x, no model consistently outperformed PyTorch, and 28 percent of the best model’s correct kernels increased peak GPU memory. NVIDIA’s SOL ExecBench pushes the standard further still: it checks submissions for reward hacking and ranks them by a SOL-Score, which grades a kernel against the analytically derived roofline of a B200 rather than against a software baseline. The state of the evidence, then: models can write working kernels, and any claimed speedup is only as trustworthy as the baseline configuration and input distribution behind it. When you read the next result in this space, those are the two things to check first.

Source

Benchmark design and results from [KernelBench](https://proceedings.mlr.press/v267/ouyang25a.html) and [KernelBench-Verified](https://arxiv.org/html/2607.16241); speed-of-light grading from the [SOL ExecBench](https://github.com/nvidia/sol-execbench) README. Follow-ups are collected under [AI-generated kernels](https://learn-kernels.com/chapters/reading#ai-generated-kernels) in the reading list.