From “viewing videos generated by AI” to “playing a world created by AI with one’s own hands” – what lies between them is actually a barrier related to “real-time”.
On October 15, 2026, Kunlun Wanyi officially released the world’s first video large-scale model game running on a single consumer-grade graphics card—Matrix-Game 4.0. Matrix-Game 4.0 established a new paradigm for game world modeling: the game engine maintains the world state and interaction rules, while the video generation model renders detailed visuals in real time. It brings a real-time gaming experience of 480p and 60 FPS into a single NVIDIA RTX 5090; from your press of a button to the image reacting, the latency is less than 150 milliseconds. Matrix-Game 4.0 has successfully completed the full closed loop of “code generation for game logic → structural visuals providing structural constraints → real-time rendering of video models”, enabling a truly implementable technical path for game prototype development, personalized customization, and UGC. The project has been opened source.
Open Source Address
Huggingface: https://huggingface.co/Skywork/Matrix-Game-4.0
1. From a piece of code to a playable world—the four fundamental pillars of Matrix-Game 4.0
1. Single-card real-time: Get “playable video models” out of the lab
Generating game images in real time on consumer-grade graphics cards was once a hard-to-achieve goal—video generation models require substantial computing power, which has long been confined to multi-card clusters in laboratories.
The answer provided by Matrix-Game 4.0 is a set of hard metrics: a single NVIDIA RTX 5090, with the game running at 60 FPS at 480p resolution; the end-to-end latency is less than 150 milliseconds from player input to first frame response; the steady-state generation speed for single-step denoising reaches 83 FPS, and the memory usage peaks at 18.5 GiB.
This set of numbers means that video-model-driven games have finally gained real value for local deployment and real-time interaction—it is no longer an offline rendered demonstration video, but a game you can actually play on your computer.
2.Harness New Paradigm: Making the World State Really “Managed”
Matrix-Game 4.0 has redefined the relationship between the video model and the world: The Coding Agent builds lightweight games through code, while the engine acts as a “referee” to maintain the world state, enforce interaction rules, and produce structured visuals and dynamic semantic conditions; the video model serves as a “painter” responsible for transforming simple geometry into rich visual effects.
External constraints and continuous calibration of the engine allow the world state to be maintained continuously, the rules of the game to be precisely defined, and gameplay to be rewritten in natural language—this is a question of “Is the world being managed seriously?” that the Genie paradigm has never truly addressed.
3. Integration of data, training, and runtime: a reusable engineering pipeline
From structure images - detailed image data pairs collection and dynamic semantic annotation, to controllable video training, few-step auto-regressive distillation and real-time reasoning, unified geometric and semantic conditions cover the entire technical chain.
This process provides an engineering foundation for expanding new scenarios, new interactions, and new gameplay methods: it reduces the dependence on detailed 3D asset creation, enables rapid development and iteration of game prototypes, and turns the slogan “small teams can build a world” into reality.
4. Publish synchronized open source: Let creators truly "play" with it
Matrix-Game 4.0 was released openly, providing researchers, developers, and creators with reusable technical foundations and adaptable game code. Combined with Coding Agent, creators can adjust scenarios, customize interactions, and design gameplay using natural language—lowering the barriers to research and creation to the level of a single Prompt, enabling more ideas to become games that can be experienced personally and continuously improved.
II. Three fundamental obstacles: Why a world that “can be walked” is “not fun”
Looking back at the previous generation of interactive world models, the problem never lay in image quality – they were already able to generate visually pleasing scenes. The real bottlenecks lie at a deeper level. When analyzing the previous generation’s approach, there are three problems that never got resolved.
1. The world state lacks an independent, verifiable maintenance mechanism
Without external engines maintaining explicit state, the continued existence of scenes and objects primarily relies on the internal representation and historical memory of the model. The model can “remember” images it has seen before, but this does not mean that developers can accurately read, modify, and verify the state of every object, nor is it possible to ensure that characters and events outside the view range always evolve consistently according to specified rules.
2. The consistency of long-term interaction still remains difficult to stabilize and ensure
Recurrent generation depends on previous generation results, and local deviations may spread and accumulate with continuous interactions. When players explore for a long time, repeatedly return to the same location, or have objects enter and exit their view multiple times, the scene structure, object identities, and interaction outcomes may drift. Expanding the context and introducing a memory mechanism can improve these issues, but it does not ensure long-term consistency at the game level.
3. A reasonable interactive interface does not equal reliable game rules
The model can generate visual effects such as door opening, collisions, or attacks, but a complete game also requires precise numerical calculations—life value changes, item ownership, task progress, and win/loss determination. These mechanisms must not only “look reasonable,” but also be clearly defined, modified, and tested by developers, and remain consistent across different operations and long-term gameplay.
Three questions, actually pointing to the same thing: the existing approach leaves “the world” stored in the model’s memory, while what a game needs is a world that can be read, modified, and verified.
This is the starting point for Matrix-Game 4.0 to adopt the paradigm of “building games with code and real-time AI-generated visuals”—instead of having the model try to remember the entire world, it’s better to provide the world with a clear and maintainable medium.
III. New Paradigm: Let the engine rule the world, let AI paint the world
The solution for Matrix-Game 4.0 is to attach a “Harness” to the video model that is responsible for maintaining the world state—the external game manager provides spatial layout references, manages the world state, and updates the semantic dynamics. The video model uniformly receives these context and control conditions, and generates short-term, realistic future scenes autoregressively.
The entire system consists of three modules, with clear divisions of labor:
The engine is responsible for recording what exactly happens in the world, and determining the outcomes of the player’s actions. It does not need to handle the full detailed visual rendering; instead, through simplified geometric models and scene structures, it describes roads, buildings, characters, and interactive objects. Status and rules such as health, backpacks, gravity, collisions, and task progress are all clearly maintained and executed by code, rather than being inferred from the generated image. As the player operates, the engine continuously updates the world and generates structural images, providing constraints for the video model in terms of scene layout, object movement, and camera perspective. Even when an object leaves the view, its state is still maintained by the engine according to the game rules.
The programming AI is responsible for transforming the player’s ideas into executable games. The player can express their requests in natural language, from adjusting the terrain and arranging objects, to modifying NPC behavior and designing new gameplay mechanics. For example, “having monsters rush in from all directions” not only means having more monsters on screen, but also requires implementing the generation locations, movement methods, attack behaviors, and winning conditions. Programming agents can also help define the semantics and visual style of the scene: the same simplified building can be set as an old house by the sea, or as a store in a future city—these descriptions serve as text conditions that guide visual generation along with the structural image.
The video model is responsible for transforming structural scenes into a detailed visual experience: while preserving the scene structure and motion constraints, it generates materials, lighting, ambient atmosphere, and rich visual details, reducing the need to create individual detailed 3D assets. It combines current control conditions and historical scenes, generates continuously in a causal autoregressive manner, and optimizes through a few-step distillation and inference for real-time re-rendering, thereby reducing the wait time between player action and image response.
Real-time execution process: Player operation → Engine updates world state → Output structure image and control conditions → Video model rendered in real time → Presented to the player
The key to this approach is to separate how a game runs from how it is presented: the code carries the rules, the engine maintains the state, and the video model generates the visual appearance. By combining the creative capabilities of programming agents, Matrix-Game 4.0 enables more people to transform their ideas into playable, modifiable, and continuously evolving games with a lower production threshold.
4. From data to inference acceleration: a complete connected chain
A "game world" that can run in real time, backed by the integration of three capabilities: data, training, and runtime.
Data: Structural images built for the world model – detailed image alignment dataset
The team built a structural image - detailed image cross-domain alignment dataset for world models and video generation using Unreal Engine 5 (UE5). This dataset covers four typical scenarios: the large city of CitySample, open desert, complex castle, and high-dynamic combat.
A self-developed lightweight geometric abstraction collection pipeline, while maintaining strict spatiotemporal synchronization between camera trajectories and timestamps, abstracts complex scenes into low-poly structural representations that preserve spatial topology, rigid pose, and dynamic trajectory, and aligns them pixel-by-pixel with high-poly cinematic images; at the same time, it captures object instance IDs, visibility states, and causal interaction events in parallel, constructing a richly semantic dynamic metadata stream.
Dynamic Caption: Let the description follow the game
In interactive games, the screen changes as the player operates: different objects are visible when the view angle is changed, and the object states also change after movement or event triggers. Fixed text descriptions are difficult to reflect these changes in time.
Matrix-Game 4.0 introduces dynamic Caption: The system associates the ID of objects in the scene with semantic descriptions, and retrieves the current visible objects and real-time states from the game engine, filling in the prompt template during runtime. The structural image provides structural information such as position, layout, and motion, while Caption adds semantic information like “what objects are present in the image and in what states they are,” and both together constrain the generation.
Controllable Generation and Spatial Memory
The video model is modified based on Wan2.2 5B to a Video-to-Video (V2V) mode: the structural images rendered by the engine are concatenated with the video to be generated per channel (148 channel input), telling the model “what to draw and where to draw it”; at the same time, a prompt for dynamic synthesis scene events, a spatial memory reference map, and the first frame are input, which are responsible for detailed description, scene consistency, and image continuity respectively.
Rendering model network node: 148 channel拼接 input, 30-layer causal DiT, self-attention K/V cache, cross-attention text K/V
Reasoning acceleration: extract every millisecond
To make 60 FPS truly achievable on consumer-grade graphics cards, the team carried out extensive engineering optimizations: using CUDA Graph to pre-record the main DiT computation and replay it all at once during runtime; writing specialized GPU kernels for RoPE, normalization, and FP8 quantization, allowing the same input quantization to be used for Q, K, and V in the same layer; the text condition KV is recalculated only when the caption is updated; the video history KV uses a ring buffer instead; self-attention is computed in parallel after being divided into KV windows; and the encoding/decoding algorithm switches to the lighter TAEHV. Finally, the text encoder, DiT, and VAE can all be running permanently on a RTX 5090.
5. Coding Agent: With a single idea, rewrite a game
Matrix-Game 4.0 is perhaps not as exciting because of how well it is drawn, but rather because it is very easy to “modify”.
With the Coding Agent, players can customize their own games using natural language based on existing game code: from adjusting the terrain and arranging scene objects, to modifying the behavior and interaction logic of NPCs, and even designing entirely new gameplay systems.
Creativity is no longer stuck on the 3D asset production line; it can directly enter the “playable and iterative” stage.
This is also the practical significance of this technical path: as AI programming assistants advance rapidly, the cost of building lightweight game engines has been significantly reduced. We also encourage creators to continue adapting and expanding the game code provided by projects, and to jointly explore more ways of playing.
Over the past year, video generation models have achieved many new results in games – but whether they can run on ordinary players’ computers is another matter.
From this comparison chart, in terms of end-to-end latency, Code World Model is approximately 10.2 seconds, StreamRender-H3 is about 545 milliseconds, AgentGarten is around 878 milliseconds, and Matrix-Game 4.0 has only 112 milliseconds—from pressing a key to the image responding, it is approximately 1/90 of the former, 1/5 of the latter, and 1/8 of the third, marking the first time it truly falls into the “real-time playable” range.
In terms of computing power requirements, other solutions require 1× H200 141GB, 8× GB200, and 1× H100 data center cards respectively, while Matrix-Game 4.0 runs only on a 1× RTX 5090 – a consumer-grade graphics card with 32GB of video memory is sufficient.
In terms of open source, Code World Model and StreamRender-H3 are fully open, AgentGarten is partially open, and Matrix-Game 4.0 is fully open in terms of code, weights, and white model synchronization.
6. Open source, letting more people join in and "play" together
Matrix-Game 4.0 was released open-source at the same time, providing researchers, developers, and creators with reusable technical foundations and adaptable game code. We also encourage creators to continue adapting and expanding the game code provided by the project, so we can explore more gameplay options together.
In the future, the team will continue to iterate the model: deploying the refiner to improve image resolution, exploring a consistent multiplayer experience from multiple players' perspectives, providing finer-grained visual control, and expanding support for more game types and scenarios.
From “controllable videos” to “operable worlds”, this step may be more important than it seems. It means that the threshold for creativity is lowering, and the boundaries of imagination are opening up.