TL;DR: The 11x latency spread in LLM gateway benchmarks comes from backend selection and cache policy, not from the gateway proxy itself. The proxy layer is thin, sometimes faster than direct calls because of connection reuse. If you benchmark only TTFT, you will pick the wrong gateway and miss the actual lever for production speed.

Key Takeaways: - The gateway layer is the visibility plane, not the bottleneck. The proxy itself rarely sets the spread. - Single-number benchmarks hide the reasoning-model cliff, where TTFT jumps from ~1s to several seconds. - Backend choice (Cerebras vs Berget) creates a ~10x throughput gap that no proxy can close. - Prefix caching reclaims 20-50% of latency by turn 6 in multi-turn chats, even when the API says no cache hits. - Production benchmarks need five metrics, not one, and a reproducible harness to be honest.

Most platform engineers assume LLM gateways are latency liabilities. The Opper benchmark of 14 gateways found the opposite. The gateway layer is thin. The 11x spread comes from choices that have nothing to do with the proxy itself.

The Opper benchmark, which drove the public "11x" claim, showed OpenRouter at 0.640s time-to-first-token versus OpenAI direct at 0.712s. The gateway was 70ms faster, not slower. That happens because connection reuse and warm TLS sessions at the proxy edge skip the cold-start cost. You pay that cost on a raw API call. Opper, another gateway, sat inside confidence intervals of OpenAI direct. The proxy is thin. The variation is elsewhere.

That changes how you should think about gateways. They are not the bottleneck. They are the visibility layer. You can see latency, route around it, and enforce policy across backends. The 11x spread lives in the choices the gateway helps you make, not in the hop through it.

The five dimensions that actually matter for LLM inference performance are: - Time to first token (TTFT) - delay before the first answer byte - End-to-end request latency - total wall-clock for the full response - Intertoken latency - gap between successive generated tokens - Tokens per second - throughput during streaming - Requests per second - concurrency the backend can sustain

If you only watch one of these, you miss the others. The spread between them is where the 11x hides. If the gateway layer is thin, where does the 11x swing come from? Most engineers measure the wrong dimension entirely.

Average latency is a lie that everyone agrees to tell. Reasoning models push TTFT from around one second for non-reasoning models to several seconds or more. When you average those two distributions into one number, you get a mean no production workload ever sees.

The mechanism is structural. Reasoning models generate internal chain-of-thought tokens before the first answer token. The user sees nothing during that hidden phase, but the clock runs. A benchmark that measures "first visible token" undercounts the cost of reasoning models. A benchmark that measures "first answer token" including the chain-of-thought produces a number that looks catastrophic. Both can be "correct." Neither tells you what production will feel like.

This is why time-to-first-token diverges from end-to-end latency in published numbers. A reasoning model might have a TTFT several seconds longer than a non-reasoning model. It can still finish the full answer faster, because most of the work happened before streaming started. A non-reasoning model might have a fast TTFT but stream for much longer. The average "latency" can be similar. The user experience is completely different.

The real production killer is multi-turn chat, and it hides deeper. Cache effects only appear by turn 6, when most benchmarks have already stopped measuring. The Opper test showed that by turn 6, most backends ran 20-50% faster than without caching. That held even when APIs didn't report cache hits. If your benchmark only sends a single prompt, you never see this. The transformer attention mechanics behind that cache effect explain why no single-number claim is safe.

Once you accept that "latency" is a vector, not a scalar, a different culprit emerges for the 11x spread.

The variation lives in the backend. Cerebras delivered 1,667 tokens per second against Berget's 174 tokens per second for the identical gpt-oss-120b model. That is about 10x throughput on the same model with the same prompt. No gateway, no matter how clever, can recover that gap. The tokens are produced server-side. The proxy is just a messenger.

This is the lever. Gateway choice affects routing, auth, observability, and caching policy. Backend choice affects raw speed. Most teams spend weeks comparing gateway dashboards and never run the same prompt against two backends. That is the wrong direction.

Semantic caching closes the gap in multi-turn workloads by reusing prefix computations. The KV cache behavior in production reuses earlier attention work, even when APIs do not report cache hits. A gateway that manages prefix caching compresses the backend's inference cost across providers. A naive gateway strips or rewrites the prompt. That forces the backend to recompute attention keys on every turn. This happens even when the conversation prefix is identical. That is wasted GPU time, paid for at the API.

The intuition that "closer means faster" is mostly wrong for inference. Regional deployment matters less than backend selection for most workloads. The dominant cost is GPU time on the inference server, not network hops. A faster backend in another region will often beat a slower one in your region once network overhead is accounted for.

If backend and cache strategy dominate, the gateway you pick matters for orchestration features, not raw proxy overhead. That changes what you should actually benchmark.

If you measure only TTFT, you will pick the wrong gateway and miss the real optimization. These five metrics predict production behavior. Each one catches a different failure mode.

Metric 1: p50 and p99 TTFT, per model class. Reasoning and non-reasoning models do not share a distribution. Measure them separately. A pooled p99 hides the reasoning-model cliff entirely. The separated distributions are the real signal.

Metric 2: Intertoken latency at the 75th percentile. Streaming UX breaks at the tail of the distribution, not the median. If your p75 intertoken latency drifts into ranges where users perceive stuttering, that matters more than the median. The 95th percentile is too pessimistic for day-to-day UX, and the median hides the problem.

Metric 3: Cache hit rate over a 10-turn conversation. Run a fixed 10-turn script. Record how many turns benefit from cache reuse. The number tells you whether your embedding and RAG pipeline performance is preserved by the gateway, or destroyed by prompt rewriting.

Metric 4: Failover latency. Time from primary backend error to first token on fallback. A gateway with slow failover can turn a transient backend hiccup into a user-visible outage. This is the metric that decides whether your users see an outage.

Metric 5: Cost per 1k tokens at the latency floor. The cheapest gateway is rarely the fastest. Plot cost against p99 latency. The sweet spot is rarely at either extreme.

These metrics are only useful if you can reproduce them. The harness pattern below drove the 14-gateway comparison and is small enough to extend.

A benchmark that runs once and never again is a story, not data. The harness below drives 14 gateway endpoints with identical prompts, warm connections, and percentile aggregation. It is minimal on purpose so you can extend it.

Three design choices to copy: - Pin GPT-4.1 as the model. It is available across all 14 gateways tested. That isolates the gateway variable from the model variable. Newer models skew the comparison toward whichever gateway adopted them first. - One warmup pass is mandatory. Cold-start variance produces large swings on the first request. The cause: TLS handshake, JIT warmup, and connection pool init. Discard it. Your production traffic never sees that. - Use a 10-turn variant for the cache metric. Run the harness with `turns=10` and a fixed conversation prefix. This surfaces prefix-cache behavior that single-shot calls never show.

The GPU and neural network cost dynamics behind warmup behavior explain why the second request is typically faster than the first.

Running the harness takes an afternoon. Interpreting it correctly takes longer. That is where the architectural decisions compound. Benchmark discipline baked in from day one separates the deployments that hold their SLA. The ones that lose it do so quietly, after the first traffic spike.

Teams that benchmark across the five metrics above redirect optimization away from gateway swaps. They focus on backend selection and cache policy instead.

The gateway itself barely moves. What moves is which provider you route to for which model class, and whether you preserve prefix-cache state across turns.

The gateway becomes a control plane, not a hot path. Once you accept that, capacity planning simplifies. You are not sizing proxies for throughput. You are sizing backend contracts, cache windows, and failover policies. That makes fine-tuning of routing rules a configuration exercise, not a re-architecture.

Long-term operational stability is a function of measured headroom. Teams that track intertoken p99, not just TTFT, see streaming quality that the user actually feels during a long response. TTFT dominates the opening experience. After that, streaming quality carries the rest.

This is the shift: from gateway-as-bottleneck to gateway-as-orchestrator. Once you see it, you stop shopping for faster proxies. Start shopping for backends with good KV-cache behavior. Pair that with production LLM routing patterns that preserve it across multi-turn chats. A few questions come up often about how this plays out in practice.

How much latency does an LLM gateway actually add?

The Opper benchmark showed the gateway layer is thin. OpenRouter measured 0.640s vs OpenAI direct at 0.712s, meaning the gateway was actually faster because of connection reuse. The 11x variation in the broader benchmark came from backend selection and cache policy, not proxy overhead.

Which LLM gateway had the lowest latency in the benchmark?

Bifrost and LiteLLM led the published results. The spread between the fastest and slowest backend for the same model was about 10x in throughput. That makes backend choice the larger lever for raw speed.

Do reasoning models change gateway latency benchmarks?

Yes. Reasoning models generate internal chain-of-thought tokens before the first answer token. That pushes TTFT from around one second for non-reasoning models to several seconds or more. Any benchmark that averages across model classes will mask this discontinuity and produce misleading "average latency" numbers.

How does KV-cache reuse affect gateway latency?

In multi-turn conversations, prefix caching made backends run 20-50% faster by turn 6. This held even when upstream APIs did not report cache hits. Gateways that manage prefix cache keys correctly capture this benefit. Those that do not force the backend to recompute on every turn.

What is the right way to benchmark an LLM gateway for production?

Measure p50 and p99 TTFT separately for reasoning and non-reasoning models. Track intertoken latency at the 75th percentile for streaming UX. Record cache hit rate over a 10-turn conversation. Measure failover latency from primary error to first fallback token. Compute cost per 1k tokens at the latency floor. Single-number averages hide the tradeoffs that matter in production.

Run the harness against your own stack and see where the real bottleneck hides.

Research and references cited in this article:

• LLM Evaluation: Frameworks, Metrics, and Best Practices ...
• LLM Inference Benchmarking: Fundamental Concepts
• What is LLM Evaluation: Best Frameworks, Metrics, Tools & ...
• The definitive guide to LLM evaluations
• LLM Benchmarking for Enterprise Production: How to Evaluate Models for Your Actual Use Case
• LLM Latency Benchmark by Use Cases in 2026
• 2026 LLM Latency Benchmarks: Analyzing Production Performance Across 200+ Models | The Efficient Frontier
• Latency-Aware Benchmarking of Large Language Models for Natural-Language Robot Navigation in ROS 2
• LLM Gateway vs Direct API Calls: Benchmarking Latency & Uptime
• Predicted-Latency Based Scheduling for LLMs
• how - Wiktionary, the free dictionary
• HOW Definition & Meaning