At the 2026 Autumn CIO and Digital Science Company Executives Summit (19th Edition) hosted by Enterprise Network D1net, Tang Anbo, Director of Silicon-Based Flow Solutions, attended the summit and delivered a keynote speech: “From ‘Computing Power’ to ‘Token Productivity’: The Inference Infrastructure and ROI Practices for the Large Model Implementation in Central State Enterprises.” Below is a summary of the key points from the speech.
At present, AI applications are in the early stage of explosive growth, and the consumption of Tokens is increasing exponentially. Behind this, the model architecture continues to evolve, with rapid version updates; at the same time, the computing power architecture is also constantly improving. The models and computing power are an iterative process that drives each other. For enterprises, on one hand, they need to adapt to these changes, and on the other hand, they must provide diverse model services for their own business scenarios while keeping costs within a reasonable range.
What Silicon-based Flow does is establish a conversion layer between computing power and Tokens: it integrates computing power under one management; through self-developed reasoning engines, it makes models run faster and better. We build Token factories on public cloud platforms, and also help central state-owned enterprises create their own Token factories internally.
In short, through systematic reasoning engineering, multiple adaptations between models and computing power are achieved. Based on this, the silicon-based flow proposes the core formula for Token productivity: Token Productivity = Capacity × Utilization Rate × Delivery Quality. Here, capacity is determined by both chip hardware capabilities and model optimization levels; utilization rate is achieved through traffic management and intelligent computing power scheduling; delivery quality includes model reasoning accuracy, task execution success rates of tool calls, and the SLA service stability required in a production environment.
From unified management to model operation, truly improve the computing power utilization rate.
From the perspectives of business and traffic, we designed a four-layer decoupled architecture. This connects model operation, model maintenance, inference execution, and computing power scheduling, covering all stages of a model’s life cycle—from deployment and acceleration to service scheduling. The layered decoupled architecture is also designed to adapt to the existing platform construction of central and state-owned enterprises, avoiding redundant construction. If an enterprise already has in-house capabilities, it can integrate with existing systems instead of starting from scratch; thus, the required capabilities can be provided directly through a “capability combination” approach.
Model gateway: not only an entrance, but also a model operation platform.
This layer is directly accessible by users of model services, such as various molecular companies and their subsidiaries. All large model calls are processed through this layer for metering, billing, authentication, and other basic tasks. More importantly, it provides a variety of scheduling and routing strategies: for example, it can route requests to the appropriate model service based on the required response time for different business scenarios, ensuring that various parties do not interfere with each other and guaranteeing satisfactory SLA levels; it also supports flexible scheduling according to different objectives such as performance priority or cost priority.
Model Service Platform: Deliver best practices for deployment and operation to enterprises.
This layer mainly manages capabilities such as model deployment and model repositories from an operations perspective. We incorporate best practices for model deployment, enabling enterprises to bring up models with a single click, rather than having to adapt them every time. We have already adapted nearly a hundred models, and new mainstream models are being adapted continuously. The goal is to make the business team able to use the models immediately after installation.
High-performance inference framework: Iterative optimization for specific models and hardware.
We have established deep cooperation with various computing power manufacturers. Focusing on specific models and computing power, we have developed our own operator library and communication library, in order to maximize the performance of the models on the corresponding hardware. For example, in collaboration with Huawei Cloud, they launched the DeepSeek R1/V3 Token service based on domestic chips for the first time in the industry. They also jointly developed and optimized inference solutions for the CloudMatrix hyper-node cluster, achieving a single-card Decode throughput of over 1,920 tokens/s on DeepSeek-R1. Globally, the commercial service deployment of the Kimi-K2 large model based on the MuXi C550 cluster was launched for the first time. On the Moore Thread MTT S5000 chip, DeepSeek-V3 was deployed, achieving a single-card Prefill throughput of over 4000 tokens/s and a Decode throughput of over 1000 tokens/s.
At the same time, our reasoning techniques are also continuously evolving: as the context of the model’s processing tasks grows and concurrency improves, it is necessary to continuously optimize the reasoning framework, such as operator optimization, communication optimization, and model quantization. Additionally, methods like PD separation and KV Cache hierarchical unloading are used to comprehensively enhance the overall performance and throughput of model reasoning.
Computing power scheduling: Flexible management of computing power resources.
Elastic and flexible computing power scheduling is a key element in reducing inference costs and improving the utilization of computing power. However, computing power planning often faces a dilemma: being too conservative may quickly lead to full load or even overload; configuring entirely at the peak during the day will result in resource waste at other times. Therefore, both elastic scaling strategies and operational methods are needed.
For example, when central state-owned enterprises provide model services to their subsidiary companies, time-based pricing can be used in internal settlements: regular pricing during the day, and discounts at night, to encourage some operations to operate at night. Each operation may use different models, which requires the platform to dynamically adjust model instances based on various metrics such as request queues and performance.
State-owned enterprise case: shifting from “renting machines” to “calling model services”
We served a central state-owned enterprise group. Initially, there were about 50 machines that provided services to subordinate subsidiaries. The early service method was very direct: each subsidiary applied for the machines and deployed the models they needed. In other words, it was like “renting machines”.
In this case, four or fifty model instances were launched, but the overall computing power utilization was very low. Some units only deployed one model after being occupied by the machine; some were used during the day and not at night; others were used only a few days a week. The machines were occupied, but no continuous effective load was generated.
We collaborate deeply with our customers to restructure the above situation: Unified models, unified computing power. Reduce duplicate models to about ten; replace the resource allocation by machine-based methods with Serverless model services. Everyone will call the API uniformly, without worrying about where the models are deployed or which machine they occupy.
In this way, with just about 50 machines, more tasks can be carried out. For example, after using some of the machines to host existing small models, multiple machines can be freed up to deploy new large-parameter models such as GLM-5.2, DeepSeek-V4-Flash-0731, etc. In the past, such models could not run within the group. But in fact, new large-parameter models are very important for business exploration, because the upper limit of model intelligence also determines the upper limit of application innovation. Through model convergence and intensive computing management, we deployed these models, and combined with performance optimization, more scenarios have changed from “unavailable” to “available”, and from “explicable” to “scalable”.
It turns out that the biggest concern on the business side is: it’s fine to develop exploration and create an Agent, but can it be scaled up for deployment? Can it handle more calls? There are questions in everyone’s mind. After the new model was launched, in tests from single concurrency to higher concurrency, the overall output speed remained above 30 tokens/s. Taking programming as an example, exceeding 30 TPS is a basic requirement for getting a good user experience. In comparative tests of the same model, we achieved a performance improvement of 40% to 100%. Of course, it depends on different input and output lengths, and the specific benefits are not the same.
Overall, first is the improved throughput of the model, and second is the increased computing power utilization. When these two factors are combined, they are reflected in the cost per million tokens, which is a metric that many companies use to measure the efficiency of reasoning services. In this case, the cost per million tokens of an original Qwen model was over 100 yuan, but after adjustment, it dropped below 10 yuan. The cost reduction is significant, but most importantly, it has opened up more business scenarios that were previously difficult to implement.
Further enhance ROI by optimizing applications and infrastructure together
As the Agent is scaled up, we have also observed a problem: there is a certain disconnect between the application layer and the infrastructure layer. In the past, when developing applications, even if they were mainly focused on the frontend, developers usually had some understanding of database indexes and SQL optimization. Now, with the clear separation between large model applications and infrastructure, the application side may not fully understand the underlying reasoning and caching mechanisms, which directly affects efficiency.
Taking coding as an example, we are working with customers to explore optimizations in this area. The software engineering of central state-owned enterprises isn’t about writing an app today and conducting another demonstration tomorrow; it involves continuous iteration over many years. The code repository is constantly updated, and submitting code and performing builds every day are part of the long-term project. How these codes are structured as the Context of the model is very important.
If the Context construction is not done well, every request may not hit the cache, and the input of hundreds of thousands of Tokens would consume a large amount of computing power. However, if it is constructed well, about 90% of the content may hit the cache, and the efficiency gap becomes evident. For example, when the cache does not hit, the system may support 100 concurrent requests; after increasing the cache hit rate, it may support 200 concurrent requests. This improvement does not require additional hardware, but reduces redundant calculations. Therefore, the Context construction itself needs to be optimized for reasoning.
In another direction, optimization can be achieved by integrating DevOps. The use of computing power has peaks and valleys; during the day, everyone uses computing power, and the input for coding tasks is long. If all code is processed only when requests are made during the day and then cached, it will consume a lot of computing power during peak hours. So, could we, during the daily build process at night, use engineering methods to preprocess the model against the code repository and store the corresponding results in a KV Cache in advance? This way, no repeated calculations are needed during the day. All these approaches have the potential to further improve the utilization rate of computing power. However, this capability also depends on our joint exploration with the scenario team.
Silicon-based flow hopes to work with central state-owned enterprises to explore more AI implementation scenarios and unleash real computing power and production capacity.
Leifeng Net (Public Account: Leifeng Net) Leifeng Net