Skip to content

CI performance regression gate

A slow kernel fails the PR.
An outage never does.

Kernalin’s GitHub Action benchmarks every pull request against main in the same environment and fails the check only on a real, statistically supported regression. It never fails your build because a service was down.

Get started Watch it work

A CI gate for kernel performance regressions

Kernalin benchmarks every pull request against the last passing build on main, and an outage on Kernalin's side never fails your build. Real example: PR #3 on JashomTech/kernalin-demo passed the Kernalin gate and was merged on 2026-10-09. Coming soon: one-click setup that opens a pull request adding .github/workflows/kernalin.yml, a verdict comment on the pull request that blocks a real slowdown (sample data), and accepting an intended slowdown with a recorded reason.

Coming soonIncludes Add CI gate (bot pull request) (7–11 s), PR comment and blocked merge (11–16 s), Accept as intended (new UI) (15–19 s), labelled in the video. Everything else is recorded from kerneln.com.

5 live · 3 coming soon · every feature below

What is a CI regression gate?

A CI regression gate benchmarks each pull request and blocks the merge if it makes performance measurably worse. Kernalin’s gate is the rnx-benchmark GitHub Action: it uploads the run, Kernalin compares it with the newest baseline on the base branch in the same environment, and returns PASS, WARN or BLOCK.

In three steps

Ten lines of YAML, then every PR is measured.

01 / 03 · Set up

Add one step to a workflow.

This is the gate from the demo repository: benchmark kernels/attention.py on every pull request to main, and fail on a real slowdown.

The token is scoped to that repository. The config file sets the blocking threshold for noisy shared runners.

.github/workflows/kernalin.yml 
name: Kernalin benchmark gate on:   pull_request:     branches: [main] jobs:   benchmark:     runs-on: ubuntu-latest     steps:       - uses: actions/checkout@v4       - name: Benchmark and gate         uses: JashomTech/Kernalin/.github/actions/rnx-benchmark@prod         with:           workload: kernels/attention.py           runtimes: pytorch_eager           batch-sizes: "1,8"           dtypes: float32           kernalin-token: ${{ secrets.KERNALIN_TOKEN }}
The demo repository's current workflow, trimmed to the Kernalin step (the PyTorch install step is omitted).

02 / 03 · Gate

Every pull request, checked independently.

On 9 October 2026 Kernalin opened pull request #3 with a verified 7.64× rewrite. The repository’s own gate benchmarked it again, passed it a minute later, and it was merged.

The optimizer and the gate don’t trust each other’s numbers. Both measure.

github.com/JashomTech/kernalin-demo/pull/3
open

Optimize attention.py: 7.64× faster (Kernalin run #73) #3

kernalin/optimize-73 → main · 1 commit · 6e4c9a0

Kernalin rewrote this kernel and verified the result before proposing it.

Speedup: 7.64× on the CPU sandbox (best median 3.058 ms).

Audit table posted on the pull request
RewriteDecisionCorrectAnti-cheatSpeedup
#1KEEPyesclean7.64×
#2REVERTyesclean0.369×
#3REVERTyesclean0.4×

Your CI's Kernalin benchmark gate checks this pull request independently.

Kernalin benchmark gate rnx-benchmark · pytorch_eager · batch 1, 8 · GitHub-hosted runner passed
All checks have passed Merge pull request
Pull request #3 opened by Kernalin on 9 October 2026 at 18:24 UTC; the Kernalin benchmark gate passed at 18:25 UTC.

03 / 03 · Decide

Strict about slowdowns, relaxed about outages.

The gate fails for the things that should fail a build: a credible regression or a broken runtime. Everything else is a warning you can read.

rnx-benchmark · outcomes

Fails the check

  • Credible regression vs baseline Bayesian verdict, same environment block
  • A runtime breaks or diverges wrong output is never "fast" block

Never fails the check

  • Kernalin unreachable or upload refused becomes a warning warn
  • Intended slowdown, accepted with a reason becomes the new baseline pass

On the pull request

  • "Add CI gate" bot pull request adds .github/workflows/kernalin.yml for you Coming soon
  • Verdict as a sticky PR comment code-owner accept, soft-fail, kernalin.yml thresholds Coming soon
How the gate decides. Live, except where marked.

Everything in it

What ci regression gate includes

Live features are in the app today. Coming soon means built and tested, waiting for the next release. GPU verification pending means no result is published until it has run on GPUs.
  • Live

    GitHub Actions regression gate

    The rnx-benchmark action benchmarks each PR against the newest baseline on the base branch in the same environment; the server re-gates every upload. Tiers are PASS, WARN and BLOCK.

  • Live

    Fails on regressions, never on outages

    A real slowdown or a broken runtime fails the build. Kernalin being down, or refusing an upload, becomes a warning instead. Report-only mode is available.

  • Live

    Accept an intended slowdown

    When a slowdown is deliberate, accept it with a reason and it becomes the new baseline.

  • Live

    Scoped CI tokens

    Each token acts for one workspace, is scoped to one repository and its baseline branches, and is stored hashed.

  • Live

    GitHub integration and auto-sync

    Connect GitHub (or GitHub Enterprise Server), link a file on a branch, and every new commit of it is profiled and benchmarked automatically.

  • Coming soon

    "Add CI gate" bot pull request

    One click opens a pull request that adds .github/workflows/kernalin.yml to your repository, instead of writing YAML by hand.

    Coming soon: Built and tested; arrives with the next app release.

  • Coming soon

    Verdict on the pull request

    One sticky PR comment and a commit status; code owners can accept a slowdown with an audited reason; soft-fail mode and thresholds as code in kernalin.yml.

    Coming soon: Built and tested; arrives with the next app release.

  • Coming soon

    GitHub App

    Per-repository install with fine-grained tokens and webhooks; follows renamed repos and branches and handles Git LFS.

    Coming soon: Built; switched on once the App is configured.

How it works

Step by step.

  1. 1

    Create a CI token

    In the app, create a token for the repository. It acts for one workspace, is scoped to that repo and its baseline branches, and is stored hashed. Add it as the KERNALIN_TOKEN secret.

  2. 2

    Add the action

    Add rnx-benchmark to a workflow with the workload path, runtimes, batch sizes and dtypes.

  3. 3

    Every PR is benchmarked

    The run is uploaded and compared with the newest baseline on the base branch in the same environment. The server re-gates every upload.

  4. 4

    PASS, WARN or BLOCK

    A real slowdown or a broken runtime fails the check. If Kernalin is unreachable or refuses the upload, the check warns instead.

  5. 5

    Accept intended slowdowns

    When a slowdown is the point (more accuracy, a new feature), accept it with a reason and it becomes the new baseline.

Honest limits

What it doesn’t do (yet).

  • GitHub Actions today. Other CI systems would need to call the same API; there is no packaged integration for them yet.
  • Shared GitHub-hosted runners are noisy. Loosen the blocking threshold in the gate’s config file (the demo repo blocks at 25%) and still see smaller moves reported.
  • The "Add CI gate" bot pull request, a sticky PR comment with code-owner acceptance, and a GitHub App are coming soon.

Recorded results

Numbers with a date on them.

Only figures from recorded runs, with their conditions.
Recorded results with date, context and source
ResultWhat was measured
7.64× Verified speedup of an attention kernel rewritten by Claude; 1 of 3 rewrites keptCPU sandbox on kerneln.com, PyTorch attention, optimize run #73, shipped as PR #3 in JashomTech/kernalin-demoSource: Kernalin optimize run #73 and PR #3 ·

Watch

A CI gate for kernel performance regressions

0:21 · captions on · recorded on kerneln.com

Kernalin benchmarks every pull request against the last passing build on main, and an outage on Kernalin's side never fails your build. Real example: PR #3 on JashomTech/kernalin-demo passed the Kernalin gate and was merged on 2026-10-09. Coming soon: one-click setup that opens a pull request adding .github/workflows/kernalin.yml, a verdict comment on the pull request that blocks a real slowdown (sample data), and accepting an intended slowdown with a recorded reason.

Coming soonIncludes Add CI gate (bot pull request) (7–11 s), PR comment and blocked merge (11–16 s), Accept as intended (new UI) (15–19 s), labelled in the video. Everything else is recorded from kerneln.com.

FAQ

Questions, answered directly

How does the CI regression gate work?

The rnx-benchmark GitHub Action benchmarks the pull request and uploads the run; Kernalin compares it with the newest baseline on the base branch in the same environment and returns PASS, WARN or BLOCK. A real slowdown or a broken runtime fails the check. If Kernalin is unreachable the check warns instead of failing, and an intended slowdown can be accepted as the new baseline.

Will the CI gate make my builds flaky?

It is designed not to: verdicts are statistical rather than a fixed percentage, comparisons are only made within the same environment, thresholds for shared GitHub-hosted runners can be loosened in a config file, report-only mode exists, and an outage never fails your build.

Can Kernalin add the CI workflow for me?

Coming soon. "Add CI gate" will open a pull request that adds .github/workflows/kernalin.yml for you; today you add the rnx-benchmark action to a workflow yourself (about ten lines of YAML).

Related

Point Kernalin at your slowest kernel.

Sign-up is open and self-serve. Connect GitHub or upload a file, and get a profile, a benchmark and verified rewrites, without changing how you ship.

Get started Talk to us