Skip to content

LLM inference benchmarking · coming soon

Serving metrics that match
what your users feel.

Point Kernalin at a Hugging Face model. It starts vLLM, proves the server answers correctly, then sweeps concurrency and reports TTFT, TPOT, goodput under your SLA, VRAM and the knee of the batching curve.

Get started See it work

0 live · 4 coming soon · every feature below

What is LLM serving benchmarking?

LLM serving benchmarking measures an inference server under load: time to first token (TTFT), time per output token (TPOT), inter-token and end-to-end latency at p50/p90/p99, throughput, and goodput (requests that meet a latency SLA). Kernalin’s serving benchmarks run vLLM with AIPerf or a built-in load generator.

In three steps

Start it. Prove it answers. Push it to the knee.

Serving benchmarks are built and tested, and waiting for GPU verification. This page shows how they work; it shows no results, because none have been recorded on GPUs yet.

01 / 03 · Start

No load until it answers correctly.

A serving benchmark that measures a broken server is worse than none. Kernalin starts vLLM from a pinned image and checks it end to end before the first timed request.

Serving job · vLLM v0.31.0
  • VRAM estimate refuse runs that cannot fit Coming soon
  • Start vLLM from a pinned image digest allow-listed engine args only Coming soon
  • Health check server is up Coming soon
  • /v1/models the right model is loaded Coming soon
  • Greedy smoke test it actually generates Coming soon
  • Concurrency sweep begins AIPerf 0.13.0 or built-in load Coming soon
The start-up sequence of a serving benchmark, in order.

02 / 03 · Measure

The latencies a user notices.

Throughput alone hides the experience. Each step of the sweep records how long until the first token, how fast tokens keep coming, and how many requests met your SLA.

Serving metrics · per concurrency step
  • TTFT time to first token p50·p90·p99
  • TPOT time per output token p50·p90·p99
  • ITL inter-token latency p50·p90·p99
  • E2EL end-to-end request latency p50·p90·p99
  • Goodput requests per second within your SLA req/s
  • Throughput requests and tokens per second tok/s
  • Memory peak VRAM and KV-cache peak GB
  • Start-up time cold start to first answer s
What each serving metric means. Values appear only after GPU runs are recorded.

03 / 03 · Decide

Find the knee before your users do.

Latency stays flat as concurrency rises, until it doesn’t. The knee is the most load you can serve inside your SLA: the number to size a fleet by.

Serving · vLLM · concurrency sweep
Coming soonGPU verification pending
concurrent requests → latency (p90) → your SLA knee: max load within SLA
Illustration of the batching curve and its knee. Not a measurement: serving benchmarks are not yet verified on GPUs.

Everything in it

What llm serving includes

Live features are in the app today. Coming soon means built and tested, waiting for the next release. GPU verification pending means no result is published until it has run on GPUs.
  • Coming soonGPU verification pending

    vLLM serving benchmarks

    Point at a Hugging Face model and revision; Kernalin starts vLLM, health-checks it, runs a greedy smoke test, then sweeps concurrency with AIPerf or a built-in load generator.

    Coming soon: First cut built; GPU verification is pending. vLLM only for now.

  • Coming soonGPU verification pending

    Metrics users actually feel

    TTFT, TPOT, inter-token latency and end-to-end latency at p50/p90/p99, requests and tokens per second, goodput under an SLA, start-up time, peak VRAM and KV-cache peak.

    Coming soon: Arrives with serving benchmarks.

  • Coming soonGPU verification pending

    Batching curve and its knee

    The concurrency sweep finds the knee: the most load you can serve while staying inside your latency SLA.

    Coming soon: Arrives with serving benchmarks.

  • Coming soon

    Safe by default

    vLLM pinned by image digest, allow-listed engine arguments (never --trust-remote-code), and a VRAM estimate that refuses runs that cannot fit.

    Coming soon: Arrives with serving benchmarks.

How it works

Step by step.

  1. 1

    Describe the model

    A Hugging Face model id and revision. A VRAM estimate refuses runs that cannot fit before any GPU time is spent.

  2. 2

    Start the engine safely

    vLLM v0.31.0, pinned by image digest, with allow-listed engine arguments (never --trust-remote-code), on a GPU runner with serving enabled.

  3. 3

    Prove it works first

    A health check, /v1/models, and a greedy smoke test must pass before any load is applied.

  4. 4

    Sweep concurrency

    AIPerf 0.13.0 or the built-in load generator raises concurrency step by step and records every latency metric at p50, p90 and p99.

  5. 5

    Read the knee

    Goodput under your SLA and the batching curve’s knee: the most load you can serve while staying inside it.

Honest limits

What it doesn’t do (yet).

  • vLLM only for now. There is no SGLang or TGI harness.
  • Needs a GPU runner with serving enabled.
  • No serving result has been recorded on GPUs yet, so none is published.

FAQ

Questions, answered directly

Does Kernalin benchmark LLM serving?

Coming soon. vLLM serving benchmarks (TTFT, TPOT, inter-token and end-to-end latency, goodput under an SLA, VRAM and the batching knee) are built and waiting for GPU verification. vLLM only for now; no results are published yet.

Related

Point Kernalin at your slowest kernel.

Sign-up is open and self-serve. Connect GitHub or upload a file, and get a profile, a benchmark and verified rewrites, without changing how you ship.

Get started Talk to us