> ## Content Index
> Fetch the complete content index at: https://content-blog.runvexity.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# How we benchmark private LLM endpoints
- URL: https://content-blog.runvexity.com/how-we-benchmark-private-llm-endpoints/
- Published: 2026-11-05T13:51:05.000Z
- Updated: 2026-11-05T13:51:05.000Z
- Description: Single-user speed, users served under one second (p95) and total throughput: what each number means, how we measure it, and how to read our results for your own workload.
- Author: Runvexity
- Tags: Benchmarks, Performance, Inference

Every benchmark number on our website comes from the same setup we use for managed endpoints: the same GPUs, the same inference engine and the same network path. This article explains what we measure, how we measure it, and how to read the numbers for your own workload.

## The three numbers we report

We describe performance with three metrics. Each one answers a different question about how your endpoint will behave in production.

- **Single-user speed:** how fast one request generates tokens when the GPU is not shared.
- **Users with first token under 1 s (p95):** how many people can use the endpoint at once while 95% of requests start responding within a second.
- **Total throughput:** how many tokens the configuration produces per second across all users at that load.

**173** tok/s

Single-user speed

gpt-oss-120b · 1× RTX PRO 6000

**128+** users

First token < 1 s (p95)

Qwen3.6 35B MoE · 1× RTX PRO 6000

**≥96%**

Throughput retained

Behind TLS, API keys & metering

## How we run a benchmark

Each run follows the same procedure, so results are comparable across models and configurations.

1. Deploy the model on the target configuration with production settings.
2. Warm up the engine before any measurement is recorded.
3. Ramp up concurrent users step by step, recording time to first token and generation speed for every request.
4. Report the highest load at which p95 time to first token stays under one second.

> “Averages hide the slow requests your users actually notice. That is why we report p95.”
> 
> Runvexity engineering

### Calling the endpoint

Benchmarks use the same API your application calls, so the numbers include the full request path.

```bash
curl https://<your-endpoint>/v1/chat/completions \
  -H "Authorization: Bearer $RUNVEXITY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-oss-120b", "messages": [{"role": "user", "content": "Hello"}]}'
```

![](https://content-blog.runvexity.com/content/images/2026/11/cover-1440x810.jpg)

Time to first token (p95) as concurrent users increase. Illustrative chart.

## Reading the results for your workload

Benchmarks describe specific models on specific hardware. Your prompts, context length and traffic pattern will move the numbers, which is why we test your workload on its intended hardware during the compatibility check.