Conceptual illustration, not measured performance data.
A server can produce more tokens per second while a chat window takes longer to show any response. Before approving an inference upgrade, ask a practical question: which part of the user’s wait are we trying to shorten?
Two kinds of “faster” on one scorecard
Merged on October 6, vLLM PR #60140 reports GLM-5.3 results after switching to an FP8 KV cache. With TP4, 8,000 input tokens, 1,000 output tokens, and concurrency of 1, 4, or 16, the table shows throughput gains of 2.3%–5.5%. Yet median time to first token (TTFT) rose 6.9%–11.1%. Median total latency fell 3.1%–6.3%: the answer started later but finished sooner. Source: PR #60140
These are contributor-reported results; we did not reproduce them. They do not include a sweep across longer inputs. A subsequent patch narrowed the default behavior to NVIDIA SM100. Applying these findings to every GLM deployment would exceed the evidence. Source: PR #60281
Separate the parts of the wait
Users wait for the first output, then for more content, and finally for the complete answer. Server throughput measures aggregate production; TTFT measures startup delay; inter-token latency (ITL) describes gaps between successive streamed outputs; completion time covers the entire request. vLLM’s server-side TTFT includes input processing, so it is not a measure of GPU computation alone. Network, gateway, and display overhead also deserve client-side observation. Source: metrics design
Scheduling involves tradeoffs. vLLM’s chunked prefill splits long inputs into pieces and schedules them alongside requests that are already generating. Its documentation says smaller batched-token budgets favor ITL, while larger budgets favor TTFT. This is a general explanation of how latency metrics can diverge. The PR above does not establish it as the cause of that FP8 change’s TTFT regression. Source: optimization guide
Caching also saves work in a particular phase. Prefix caching reuses computation for a shared input prefix, helping prefill without directly shortening the decode work that generates new tokens. Repeated questions about one long document can therefore benefit differently from workloads dominated by long answers. Source: prefix caching and its limits
Write acceptance criteria before upgrading
The following is our proposed testing plan, not a set of completed experiments.
-
Hold comparison conditions steady. Record the GPU, model version, inference-framework commit, quantization, parallelism, speculative decoding, and sampling settings. Change only the item under evaluation at a time so you can identify where a gain comes from.
-
Build a small matrix. Choose three real task types: short conversations, short answers about long documents, and long outputs. Test each with new prefixes and reusable prefixes, under low, normal, and near-capacity load. That makes 18 cells. Keep inputs within the model’s supported range and choose load levels from your own traffic.
-
Add a traffic burst. Record actual arrival rate, actual concurrency, queueing, and failed requests. A fixed-concurrency test and a sudden influx of users impose different pressures; a limiter may also move waiting to the client. vLLM’s benchmark tool separates request rate from maximum concurrency and exposes client-queue metrics. Source: benchmark parameters
-
Keep the whole scorecard for every cell. At minimum, record p50/p95/p99 TTFT, ITL, and total latency, plus throughput, cache hits, preemptions, timeouts, and error rate. Repeat tests independently and disclose warmup methods and sample sizes. Interpret tail percentiles especially cautiously with small samples. Quantization changes also need a separate task-quality check: speed cannot establish correctness.
-
Set thresholds before comparing gains. An interactive assistant may prioritize first output and streaming pauses; an offline batch job may prioritize completed volume. Product requirements should determine the actual thresholds. Counting successful requests that also meet latency targets gives a more useful measure of service capacity. vLLM supports goodput thresholds for TTFT, time per output token, and end-to-end latency. Source: goodput option
Connect the result to the user
A useful acceptance report should answer one concrete question: “With our traffic and quality requirements, how many more requests can we finish per hour, and how long will the slowest-served users wait?”
Higher throughput is welcome. Read it alongside waiting times, quality, and failures to see exactly what an upgrade improves.
Sources checked on October 19, 2026. This engineering analysis uses public contributor benchmarks and official documentation. The test matrix is our proposal; we performed no independent performance or quality testing. The documentation’s latest pages may change. Check the version you actually deploy.