First Token latency reduced by 32.4%! Shishi Technology's cluster scheduling big move boosts domestic inference efficiency again

Yunzhong reporting from Aofei Temple | QbitAI, WeChat public account QbitAI

In a GPU cluster, request queuing and idle resources can happen at the same time.

During the 2026 CCF National High-Performance Computing Academic Conference, Shishi Technology released a new product,the TokenFleet distributed scheduling framework,focusing its optimization on request routing, cross-datacenter capacity coordination, and mixed-workload governance, attempting to raise the effective output of existing computing power.

If inference services are viewed as a "domestic Token optimization factory", GPUs are the production equipment, inference instances are the workstations, and business requests are individual orders. What TokenFleet is responsible for is matching orders with production capacity.

With the same batch of equipment, different tasks and different allocation methods lead to different waiting times and compliant output. Serving at scale requires considering these production arrangements together.

TokenFleet focuses on the cluster's effective output

After model deployment, the cluster still needs to handle multiple types of business simultaneously.

Multiple models share computing power, and different tenants have their own service-level requirements. Short conversations need timely responses, long document processing requires more computation and video memory, and offline batch tasks have completion deadlines. When services are spread across multiple datacenters, the coordination of capacity at each location must also be considered.

Simple traffic splitting may leave some instances with long queues while capacity elsewhere is not utilized in time. Long requests and batch tasks squeezing each other slows down real-time interaction; timeouts, cancellations, and repeated computations consume resources without producing compliant deliveries.

Under the condition that model quality, service requirements, and computing power budgets are consistent, TokenFleet, throughincreasing effective Token output, lowers the cost per unit of effective output.。

TokenPilot selects instances within the same model pool, FleetMind handles cross-datacenter capacity coordination, and SLOMax is responsible for admission, queue governance, and capacity allocation for mixed workloads—the three core technologies each have their own division of labor.

Who a request is assigned to, when a task enters the queue, and how datacenter capacity is coordinated all ultimately come back to the same metric: effective Token output that completes successfully and meets service requirements.

TokenPilot calculates cache benefits together with queuing costs

First, request placement. Which instance should a request be sent to?

A short Q&A and a long document may both count as one request, yet their consumption of computation and video memory can differ greatly. Splitting traffic merely by request count makes it hard to balance the actual workload across instances.

Caching adds a layer of trade-offs to this. Requests carrying the same long prefix, if sent to an instance that retains the relevant KV cache, have the opportunity to reuse previous computation results, reducing Prefill—that is, the repeated computation when processing input content.

However, instances with high cache hit rates may also have long queues. When requests all flock to the same place, the computation time saved may be spent on waiting.

When selecting a placement, TokenPilotsimultaneously estimates cache reuse benefits and queuing costs.It estimates workload by combining input size, output budget, and reusable prefixes, then filters and scores candidate instances based on queue status, in-flight Tokens, KV cache pressure, and predicted latency. The output budget is a constraint and does not represent the actual output length.

The scheduler uses this to compare the estimated processing costs of different instances, judging whether the computation saved by reusing cache can offset the extra waiting.

According to tuning data on GLM5.3 provided by Shishi Technology, under the tested GPU cluster and workload conditions, after adopting predicted-latency routing,P95 First Token latency decreased by 32.4%, and P50 First Token latency decreased by 71.3%.。

△Table 1 TokenPilot First Token latency tuning results. Data source: Shishi Technology conference presentation materials. Improvement rate is calculated as (baseline value − optimized value) ÷ baseline value; latency reduction does not directly equal cost reduction.

P50 reflects the median level, while P95 reflects the waiting situation of tail requests. This set of data shows the response improvement under the tested conditions; the cost-benefit still needs to be evaluated in combination with throughput and resource usage.

FleetMind must also account for service capacity across data centers

When the perspective shifts from a single model pool to multiple data centers, scheduling gains an additional layer of capacity judgment.

Having idle GPUs does not necessarily mean requests can be accepted immediately. Whether the target model is deployed, whether the version matches, whether the serving endpoint is ready, and how much time network transmission will take all affect the effectiveness of traffic splitting.

What FleetMind focuses on isthe actually available model serving capacity. In the design of TokenFleet, it aggregates the model availability, real-time capacity, and network status of each data center, first selecting the executing data center, and then the local decision engine of that data center selects the specific inference instance.

When a hot data center starts queuing, the system evaluates whether other data centers have ready endpoints for the same model version and whether they still have spare serving capacity, and then determines whether new requests that allow cross-region execution can be accepted.

Whether requests can be diverted depends first on satisfying permission and data-residency constraints. Whether diversion is worthwhile also requires comprehensively comparing origin-fetch latency, queuing, compute, caching, and transport costs.

Capacity coordination also operates on different time scales: short cycles adjust where new requests go, while long cycles adjust model replicas and pre-warming locations, combined with capacity reservation, load-ceiling awareness, and automatic fallback to reduce capacity mismatch.

△Figure 3 FleetMind cross-datacenter execution framework. Note: Site A is the business entry point and is not counted in the GPU capacity pool; Site B is the existing cluster, and Sites C and D are expansion datacenters.

TokenPilot and SLOMax target the existing cluster, while FleetMind targets expansion into newly added compute datacenters. The goal of cross-datacenter coordination is to let distributed capacity that meets business constraints absorb hotspot demand, reducing capacity mismatch and duplicate reservations.

SLOMax schedules production according to service requirements

Requests enter the same cluster, yet their delivery requirements differ widely.

Real-time conversations care about first-token response and sustained generation speed, long-context tasks rely more on input-processing capability and KV capacity, and offline batch tasks focus on completion deadlines and throughput. Placing them in a single queue easily causes resource contention.

Long Prefill, KV capacity pressure, and batch workloads can all squeeze interactive requests. Timeouts, cancellations, and repeated execution also leave part of the computing unable to translate into effective delivery.

SLOMax identifies workloads using input length, output budget, task type, and tenant service objectives, then admits by tier, queues in separate pools, and coordinates capacity. It must safeguard interactive capacity and control the concurrency of long and short requests and their mutual interference.

△Figure 4 SLOMax classified-queue and capacity-feedback framework. Note: effect feedback is used to adjust capacity and admission; the framework does not presume preemption capability for running tasks.

To measure the effectiveness of this scheduling, TokenFleet uses compliant output as its optimization target.

Goodput, i.e., compliant output throughput, is computed as: the total volume of output tokens produced by requests that completed successfully and met the agreed service requirements during the entire test period, divided by the test duration. Different task categories require separately defined compliance thresholds and separately reported results.

The system feeds compliant throughput, failures, cancellations, and repeated executions back to the scheduling policy, continuously adjusting admission and capacity ratios. How busy the GPUs are and how high total throughput is must both be viewed together with the final qualified delivery.

In the GLM5.3 benefit estimation example given by METASTONE, after adopting flow-control coordinated scheduling, total output throughput increased by about 1.5%,compliant output throughput rose by about 29.4%, and the compliance ratio climbed from about 67% to about 86%.

△Table 2 SLOMax effective token throughput benefit estimation example. Data source: METASTONE conference presentation materials. This table is a benefit estimation example; the compliance ratio uses the rounded values from the original text, with a change of about 19 percentage points.

These figures belong to a benefit estimation, illustrating the output structure that scheduling aims to improve. Actual gains still need to be validated against specific workloads and service thresholds.

For the same total output, if timeouts, cancellations, and repeated computation can be reduced, there is an opportunity to deliver more compliant tokens. SLOMax incorporates these results into feedback to adjust subsequent resource allocation.

The efficiency of a domestic token optimization factory also depends on scheduling

TokenFleet pushes the optimization of inference services to the cluster level. How instance performance is allocated to different requests, and how capacity across multiple datacenters participates in serving, all require scheduling to connect resources with business requirements.

METASTONE (是石科技) is anindependent third-party full-stack compute operatorthat is not tied to any specific large model vendor, providing customers with one-stop full-stack compute delivery services coveringhardware, networking, scheduling, optimization, and operations。

Drawing on rich experience originating from a national-level computing center, its compute clusters operate with an SLA above 99.95%. Currently, METASTONE has brought under management more than 20000P+ of compute resources and operates over 10 super-intelligent computing centers, including 2 national supercomputing centers; team members have won 3 Gordon Bell Prizes and have accumulated full-stack technical service experience with ultra-large-scale clusters.

Around the domestic token optimization factory, METASTONE continuously hones its software capabilities through self-developed, self-used tools, aiming to make general-purpose, high-frequency inference services more economical and to let domestic compute deliver more capability.

The previously released Meta-Infer high-efficiency inference engine focused on improving the execution efficiency of a single inference instance. This time, TokenFleet focuses oncluster scheduling, organizing services at scale through request allocation, capacity coordination, and mixed-workload governance.

Enabling more businesses to afford tokens and use them well requires both hardware to provide compute power and software to organize that capacity.

TokenFleet has focused its further optimization direction on how each request is allocated and how each unit of capacity is used.

*This article is republished by QbitAI with authorization; the views are those of the original author alone.