Artificial intelligence large models have already demonstrated astonishing strength in static tasks such as text creation, code writing, and image generation, but how do they actually perform in fast-changing real-time interaction scenarios? Recently, NVIDIA, together with top institutions including the University of Texas at Austin (UT-Austin), officially released a brand-new real-time interactive task benchmark — Realtime-Bench. The benchmark was created to provide a precise yardstick for evaluating AI's overall performance in dynamic environments.
Core Testing Mechanism: Comprehensive Examination of Three Major Dynamic Capabilities and Clock Protocols
To comprehensively test large models' ability to cope in real real-time scenarios, Realtime-Bench has been carefully designed with 50 rich tasks covering eight categories of scenarios. During testing, the benchmark's core examines three key capabilities of the models: real-time perception, temporally adaptive reasoning, and timing of actions.
Meanwhile, in order to scientifically distinguish the specific impact of different factors on model performance, the researchers set up three different environment-advancement clock protocols in the tests. This rigorous testing architecture can effectively strip away external interference and directly probe large models' foundational performance in dynamic processing.
Evaluation Results Announced: Most Models Trapped in a Double Dilemma of Low Latency and Reasoning Ability
The researchers used the benchmark to conduct in-depth testing of 12 cutting-edge large models worldwide, and the results exposed obvious shortcomings of current mainstream AI in real-time tasks.
The data shows that in the demanding scenario where the environment advances according to actual inference time, even the best-performing GPT-6-Astra achieved a task success rate of only 44%. And under high reasoning intensity, the probability of a model achieving three complete successes was a mere 36%. This indicates that the vast majority of cutting-edge large models still struggle to find a perfect balance between extremely low latency and deep reasoning capability when facing complex real-time interactions.
In-Depth Experimental Insights: Blindly Increasing Reasoning Intensity Cannot Solve the Fundamental Problem
In addition to the overall low success rate, the study also revealed several key industry insights through further experiments:
Reasoning intensity and success rate are not linearly positively correlated: In real-time tasks, blindly increasing a model's reasoning intensity does not necessarily effectively improve the task success rate.
The trade-off between depth of thought and timing of action: In actual operation, a model must precisely balance deliberate depth of thinking against fleeting opportunities to act.
The limitations of simply reducing latency: The experimental results clearly show that simply relying on reducing inference latency cannot fundamentally solve all the real-time interaction challenges faced in reality.
The launch of Realtime-Bench not only provides the industry with a benchmark tool for evaluating large models' dynamic interaction capabilities, but also sounds an alarm for AI researchers worldwide: how to break through the bottleneck of real-time perception will become a key breakthrough direction for the next generation of large models on their way to artificial general intelligence (AGI).