On October 16, the 10th CSDI Summit China Software R&D Intelligent Innovation Technology Summit was held in Shenzhen. In the main forum, Xuan Shanming, Vice President and CTO of SenseCore, delivered a keynote speech titled "AI Infrastructure Practices and Trends in the Agentic Era," sharing SenseCore's technical explorations and practices in large-scale inference services, covering topics such as Token growth trends in the Agentic era, heterogeneous inference cluster practices, and new AI infrastructure technologies.
Xuan Shanming, Vice President and CTO of SenseCore
Xuan Shanming pointed out that as AI applications gradually move from conversational Q&A toward multi-Agent collaboration and long-task execution, Token demand is growing by a factor of a thousand, and AI infrastructure isfacing a new situation where demand explosion coexists with constraints on compute supply. In the future, the key to industry competition will not be how much compute one possesses, but how, through architectural innovation, heterogeneous collaboration, and resource scheduling, one can continuously produce high-quality Tokensat lower cost, with higher efficiency and greater stability.
From compute scale to Token capacity: AI infrastructure welcomes an efficiency transformation
Compared with traditional conversational Q&A, single-Agent tool calling, multi-Agent collaboration, and long-term, unattended execution involve more model calls, inference rounds, and context processing, with Token consumption growing by a factor of a thousand. At the same time, restricted supply of high-end chips and the mismatch between existing compute and the resource demands of inference tasks make it difficult for compute supply to expand in step. Industry data shows that in 2026 inference spending is expected to account for 55% of total AI IaaS spending, rising to 59% by 2027.
Meanwhile, different applications also have markedly different Token needs. Basic Q&A focuses more on unit cost and concurrency, while scenarios such as office Agents and customer service have higher requirements for response latency and generation speed, and high-value scenarios like scientific exploration and financial quantification place greater emphasis on result accuracy, auditability, and traceability. Metrics such as Time To First Token (TTFT), output token speed (OTPS), long-context caching, concurrency capability, and call success rate together determine the quality and cost of Token services.
Xuan Shanming said: "This means that AI infrastructure can no longer rely on a single hardware configuration to satisfy all tasks, but needs, according to the service requirements of different scenarios, to organize and schedule compute resourcesin a layered and precisely scheduled manner。. Moving from scale expansion to efficiency optimization, and from single-point hardware replacement to heterogeneous collaborationhas become an important development direction for AI infrastructure in the Agentic era."
With Prefill-Decode disaggregation at its core, building a 10,000-card-level heterogeneous inference cluster
Facing the contradiction between constraints on high-end compute supply and growing inference demand, SenseCore has explored a disaggregated inference architecture,using Prefill-Decode disaggregation and heterogeneous mixed scheduling to let different types of compute take on the tasks each is best suited for.
In large model inference, Prefill processes input Tokens and is strongly compute-intensive; Decode generates output Tokens step by step, requiring continuous reading of context caches and demanding high memory bandwidth. Prefill-Decode disaggregation decouples the two: suitable domestic compute centrally handles Prefill tasks, high-bandwidth premium GPUs handle Decode tasks, and the two stages are connected through KV Cache transfer. This solution is not simply combining different chips together, but reorganizing compute resources around computing characteristics, memory bandwidth, and task requirements. Domestic compute thereby moves from handling basic inference tasks to participating in higher-level inference services, improving the overall efficiency and cost-effectiveness of the mixed cluster while meeting the service quality requirements of different scenarios.
Based on this idea, SenseCore has continuously carried out heterogeneous inference practices at the Lingang AIDC, building a 10,000-card-level heterogeneous inference cluster. Realizing this architecture requires solving interconnection problems between different generations and types of compute, integrating networks of different specifications such as 800G, 400G, and 200G in the same cluster to ensure efficient KV Cache transfer between Prefill nodes and Decode nodes.
However, as the cluster scale expands, node failures and idle resources have become new engineering challenges. For example, in some heterogeneous configurations, 30 to 50 mid-to-low-end Prefill nodes may be needed to match the processing capacity of one Decode node. Once a Decode node fails, a large amount of Prefill resources may also sit idle, affecting overall service efficiency.
In response, SenseCore has practiced two optimization methods. One is to reserve a small amount of high-end compute in the Prefill resource pool, so that when a Decode node fails, high-end nodes can be quickly deployed as replacements; the other is to enable nodes in the cluster to dynamically switch between Prefill and Decode roles, flexibly adjusting node assignments according to real-time load and resource pressure, achieving dynamic flow and elastic scheduling of compute resources. Through these two methods, costs can be reduced while ensuring that the large-scale heterogeneous cluster continuously and stably outputs high-quality Tokens.
Building on these practices, SenseCore has further constructed asix-layer architecture systemcovering energy facilities, resource interconnection, the KV data plane, the inference runtime, service scheduling, and the delivery layer, organizing heterogeneous hardware into an inference service system that can be uniformly scheduled and run elastically.
According to Xuan Shanming, under a specific test configuration, compared with the baseline solution and with the same number of high-end flagship cards, through heterogeneous inference and resource coordination,overall Token throughput can be increased to 5 times the original.
From cross-domain Prefill-Decode disaggregation to AF disaggregation: continuously expanding the boundaries of inference architecture
On the basis of large-scale heterogeneous inference, SenseCore is also exploring technical directions such ascross-domain Prefill-Decode disaggregation and AF disaggregationto further unleash the collaborative potential of dispersed compute resources.
Cross-domain PD disaggregation aims to break through the resource boundaries of a single data center. Xuan Shanming explained that after achieving PD disaggregation within a single data center, the next step to explore is cross-domain PD disaggregation, further activating computing resources across different regions. SenseTime's SenseCore works with local governments across the country; through cross-domain collaboration, local mid- and low-tier compute can handle Prefill, then transmit the KV Cache to high-tier compute clusters in other regions to complete Decode,achieving inference orchestration across clusters, across data centers, and even across regions.
The feasibility of this direction is also closely related to the evolution of model architectures: the size of the KV Cache for 1M context has dropped in some models from nearly 100GB to about 1GB, greatly reducing the pressure of cross-domain transmission. SenseCore is also exploring the introduction of a Master Decode node (MD) and a Relay Decode node (RLD) to support inference orchestration across geographically distant locations, so that even if a region lacks complete high-tier compute capability, it can integrate compute resources of different scales and performance levels, pushing the unified computing power network from resource connection toward task collaboration and delivering higher-throughput, higher-quality Token services.
AF disaggregation further decouples the different computational tasks in the inference process. In the Decode stage, Attention is mainly responsible for processing context and extracting key information, with high requirements for memory capacity and bandwidth; FFN performs a large number of matrix multiplication operations and demands greater computing power. Traditional architectures bind the two to the same set of GPU resources, which can cause resource contention. AF disaggregation deploys Attention and FFN onto hardware better suited to their respective task characteristics, achieving independent scaling and resource optimization.
In the future, AF disaggregation is expected to bring a 150% to 200% increase in per-card throughput and a 30% to 40% reduction in operating costs, improving overall energy efficiency to more than twice its original level. The related gains still depend on the specific model architecture, hardware configuration, and system implementation, but the technical approach is consistent: continuously identify performance bottlenecks in the inference process, and improve overall system efficiency through finer-grained task decomposition and resource allocation.
From PD disaggregation to cross-domain PD disaggregation and then to AF disaggregation, the continuous evolution of future inference architectures is expected to further unlock the performance and cost advantages of inference infrastructure.
From the art of parameter tuning to systems engineering, measuring inference value by SLA and cost
Xuan Shanming believes that optimizing inference infrastructure requires returning to two core metrics: one is end-to-end service quality, that is, whether the system can meet the agreed response speed, generation efficiency, concurrency capability, and call success rate; the other is cost, that is, how many effective Tokens a unit of investment can produce while meeting the corresponding service quality requirements.
This also places higher demands on infrastructure. Domestic compute and advanced compute need continuous adaptation with different models, and the rapid iteration of models and applications further increases optimization complexity. Only by incorporating hardware configuration, model characteristics, task requirements, and scheduling strategies into a unified system can better solutions be continuously found among multi-dimensional combinations.
Currently, SenseCore is enhancing its large-scale inference service capabilities through a Token factory model. The SenseCore platform's daily average Token service volume has reached 6 trillion, and its full-year Token service scale in 2026 is expected to grow 25-fold, reaching a daily average scale of 10 trillion Tokens in the fourth quarter.
Xuan Shanming stated that what is scarce in the Agentic era is not chips, butthe systemic capability to stably, cheaply, and at a high grade continuously produce Tokenswhich encompasses a complete system of system efficiency, architectural innovation, and industry delivery capability. From technical practice to business delivery, SenseCore hopes to push inference infrastructure from an experience-dependent parameter-tuning process towardmeasurable, optimizable, and acceptable systems engineeringso that investments in computing power ultimately translate into stable Token service capacity.
This article was provided by SenseTime and reposted by QbitAI with authorization; the views belong to the original author.