How we benchmark private LLM endpoints

Every benchmark number on our website comes from the same setup we use for managed endpoints: the same GPUs, the same inference engine and the same network path. This article explains what we measure, how we measure it, and how to read the numbers for your own workload.

The three numbers we report

We describe performance with three metrics. Each one answers a different question about how your endpoint will behave in production.

• Single-user speed: how fast one request generates tokens when the GPU is not shared.
• Users with first token under 1 s (p95): how many people can use the endpoint at once while 95% of requests start responding within a second.
• Total throughput: how many tokens the configuration produces per second across all users at that load.

How we run a benchmark

Each run follows the same procedure, so results are comparable across models and configurations.

1. Deploy the model on the target configuration with production settings.
2. Warm up the engine before any measurement is recorded.
3. Ramp up concurrent users step by step, recording time to first token and generation speed for every request.
4. Report the highest load at which p95 time to first token stays under one second.

“Averages hide the slow requests your users actually notice. That is why we report p95.”

Runvexity engineering

Calling the endpoint

Benchmarks use the same API your application calls, so the numbers include the full request path.

curl https://<your-endpoint>/v1/chat/completions \
  -H "Authorization: Bearer $RUNVEXITY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Hello"}]}'

Reading the results for your workload

Benchmarks describe specific models on specific hardware. Your prompts, context length and traffic pattern will move the numbers, which is why we test your workload on its intended hardware during the compatibility check.