TL;DR: Latency cuts that look like pure product wins often inflate GPU spend. The cause is reduced batch sizes and fragmented utilization. The fix is not to stop optimizing for speed, but to optimize in ways that preserve tokens-per-GPU-hour. The math, the techniques, and the SLO conversations below protect both user experience and the bill.

Key Takeaways: - Cost per token equals your GPU-hour rate divided by realized throughput, so utilization beats raw speed every time. - Speculative decoding that halves latency but halves batch size leaves unit economics unchanged. Only batching-preserving techniques cut cost. - Negotiate SLOs as cost ceilings, not latency floors, so product and infra optimize the same variable.

Your platform team just hit a 200ms p99 latency target. Latency is down, users are happy, and finance is asking why the GPU bill doubled in three weeks. The fix that made the product faster made the infrastructure economics worse.

Most inference optimization guides treat latency as a free win. Lower numbers, same hardware, better product. The framing is wrong.

Cutting latency by 100ms often means serving more requests per second on the same fleet. That forces either more GPUs or higher per-token costs. The two outcomes are not equivalent, and one of them is the trap.

Public GPU-hour rates vary widely across clouds and instance types. That spread is the entire point. Every additional GPU is a real line item, not a rounding error. A standard Hugging Face `pipeline()` call keeps an A100 underutilized under real traffic. It processes one request sequentially while the rest of the silicon sits idle. You pay for idle capacity, and latency changes that balance in ways most teams do not model.

So why does making the model faster make the bill bigger? The answer lives in how GPUs actually earn their keep, and in the machine learning inference workloads that determine what "keep" means in production.

Decode is memory-bandwidth-bound. A naively served model leaves most of the GPU idle while you pay for all of it. Cost per token is GPU-hour price divided by tokens actually produced in that hour. Utilization is the dominant variable.

When you cut latency, you change the shape of the traffic curve. Requests complete faster, concurrency spikes, and the scheduler starts queueing rather than batching. That is where the economics flip.

Here is the core insight. A GPU emitting tokens slowly and a GPU emitting tokens rapidly cost the same hourly rate. But the faster one produces tokens at a fraction of the unit cost. The denominator went up while the numerator stayed flat.

Speculative decoding cuts latency roughly 2x. Good. But if your batch size drops because requests now arrive and finish in tighter windows, your tokens-per-GPU-hour can actually fall. The latency win becomes a cost loss.

This is the trap. The lever that improves user experience can also destroy the utilization metric that determines cost per token.

The mechanism is straightforward. Faster individual requests compress the arrival-departure window. The scheduler gets less material to pack into a batch. Lower batch size, lower tokens-per-second-per-GPU, higher cost-per-token.

At scale, the same fleet sees its tokens-per-GPU-hour fall because batch sizes shrank. The bill does not care that latency is better.

The fix requires understanding when the math flips. It is not at the same point for every workload.

Latency optimization is only free when the new latency profile lets you pack more requests into the same GPU-hours through better batching. This is the condition most teams miss, and it splits the outcome into three cases. - Case 1: Speculative decoding halves latency and halves average batch size. Cost per token is unchanged. You serve the same number of users on the same fleet. Nobody wins except the dashboard. - Case 2: Speculative decoding halves latency and batch size holds. You halve cost per request at the same throughput, or serve twice the requests at the same cost. This is the win. - Case 3: Latency drops trigger product changes. More features per session, more retries from faster timeouts, higher traffic from better conversion. This is the one that doubles your bill. It has nothing to do with the inference stack. It is a demand-side multiplier.

The SLO matters more than the raw number. A 200ms p99 target that holds batch size is cheaper than a 100ms p99 that fractures batching. The second sounds better in a dashboard but is not a marginal call.

The gap between a latency target that preserves batching and one that fractures it directly determines whether your GPU spend grows or holds.

Teams that get it right frame the problem in cost-per-million-tokens, not milliseconds. Teams that get it wrong optimize for one metric and discover the other six months later when finance flags the variance.

So the question is not "how do I make inference faster" but "how do I cut latency while keeping the GPU busy." That requires specific techniques, not generic advice.

Speculative decoding with a draft model. Deploy a small, fast model that proposes tokens. The larger model verifies them in parallel. This cuts decode latency roughly 2x without reducing batch size. The scheduler still sees the same number of in-flight requests.

Chunked prefill. Configure `--chunked-prefill-size 512` in SGLang to prevent long prompts from dominating the GPU. It also improves latency fairness across concurrent requests. The tuning depends on your traffic shape: - Mixed workloads: start with 512. This is the safe default for most teams. - Short-prompt traffic: increase to 1024. You get more prefill throughput without harming decode latency. - Long system prompts under concurrency: drop to 256. This prevents the tail from blowing up when a long prompt hits a busy GPU.

KV-cache eviction tuning. Keep the cache warm and bounded so memory-bandwidth pressure stays predictable. A stable cache means stable batch composition, which means stable utilization. Cache thrash is a hidden latency source and a utilization killer.

Tensor parallelism configuration. `--tp 2` splits a model across two GPUs, which can lower per-request latency. The cost is cross-GPU communication overhead. Only worth it when batch sizes are large enough to absorb the synchronization.

Batching discipline. Continuous batching keeps the GPU fed with mixed prefill and decode tokens. The win is not raw speed. It is keeping tokens-per-GPU-hour high while latency improves. Teams that treat batching as a scheduler afterthought lose this game by default.

These techniques are standard in the KV cache management for transformer inference playbook. They are why production deployments have shipped without latency optimization blowing up their cost model. The techniques keep the utilization math intact.

But the harder problem is not technical. It is getting product teams to accept a latency target that protects the bill.

Frame latency in terms of user outcome, not milliseconds. A 200ms p99 that holds a healthy batch costs less per user than a 100ms p99 that serves requests one at a time. Product teams care about conversion and retention. They do not care about tail latency in isolation. Speak their language.

Tier your SLOs by user journey: - Interactive chat can tolerate 200-300ms p99 if time-to-first-token is under 80ms. Streaming is the win, not total latency. - Batch analytics has no business at sub-second targets. If product is asking for fast batch, the requirement is misstated. - Autocomplete and suggestion bars can run at 150-200ms p99 because users expect near-immediate feedback.

Make the cost visible. When product asks for 100ms, show the projected GPU-hour increase at current traffic, not just the latency chart. A latency improvement that inflates GPU spend needs a business case. Product should see the number before infra commits the hardware.

Set a cost ceiling, not just a latency floor. Agree that the SLO is "as fast as we can serve within $X per million tokens." This forces joint optimization rather than one-sided latency chasing. It gives both teams a variable they can defend in a planning meeting.

SLOs that survive contact with real traffic are negotiated, not imposed. Teams that treat LLM SLO cost tradeoff frameworks as a one-time architecture decision lose their gains to traffic drift within two quarters. The frameworks need to be a recurring conversation.

Mature production systems almost universally have a cost-ceiling discipline baked into their SLO process. The ones that burn out have latency-floor chasing with no financial guardrail.

When latency and cost are jointly designed instead of sequentially optimized, something changes downstream that most CTOs do not anticipate.

Cost per million tokens becomes predictable. Utilization is no longer being sacrificed to hit latency numbers, so variance drops and forecasting gets easier. Finance stops flagging the GPU line as a surprise every quarter.

GPU procurement conversations shift from "we need more hardware" to "we need the right hardware at the right utilization band." You stop buying GPUs to compensate for fragmentation. You start buying them to extend capacity at sustained high utilization. The hardware math changes shape.

Product teams iterate faster. Cost ceilings are explicit, not surprise quarterly reviews. When a new feature comes up, the conversation changes. It becomes "does this fit within our cost-per-token budget" rather than "ship it and we'll absorb the GPU cost." That discipline compounds.

The same techniques that protect utilization also let you serve traffic spikes without emergency GPU purchases. The system was designed for the spike shape, not retrofitted to it. Spikes stop being a fire drill and start being a scaling event your architecture already handles.

This is the payoff. Treating latency and cost as coupled variables, not sequential targets, is what separates systems that survive long-term from the ones that get rewritten early.

Teams building at this level treat the latency-cost tradeoff as the central design constraint, not an afterthought. The work is unglamorous, and the results compound.

Why does cutting LLM latency sometimes increase GPU costs?

Latency reductions often reduce batch sizes because requests complete faster and arrive in tighter windows. This lowers tokens-per-GPU-hour. Cost per token equals GPU-hour price divided by realized throughput, so lower utilization means more expensive tokens and a higher bill at the same traffic level.

Is speculative decoding always worth it for cost reduction?

Only when it preserves or grows batch size. Speculative decoding that halves latency but halves batch size leaves cost per token unchanged. When batch size holds, speculative decoding roughly halves cost per request or doubles throughput on the same fleet.

What is a good latency SLO for production LLM inference?

It depends on the user journey. Interactive chat tolerates 200-300ms p99 if time-to-first-token is under 80ms. Batch analytics should not carry sub-second SLOs at all. The right target is the slowest latency that does not damage conversion, not the fastest the hardware can produce.

How do you calculate true cost per token for LLM inference?

Divide your effective GPU-hour rate (including hosting, depreciation, and licensing) by the tokens per second your fleet actually produces under real traffic, not peak benchmarks. Public GPU-hour rates vary widely. Your realized rate depends entirely on utilization.

What is chunked prefill and when should I tune it?

Chunked prefill splits long prompt processing into smaller chunks. Concurrent requests then share the GPU more fairly. Start with `--chunked-prefill-size 512` for mixed workloads. Increase to 1024 for short-prompt traffic. Drop to 256 if you see latency spikes from long system prompts under concurrency.

If you want to map these techniques to your own workload, a quick audit of where batch sizes actually sit today reveals the rest.