IT之家 November 11 news — On November 10 local time, Google announced that it has partnered with AI model evaluation agency Artificial Analysis to launch the Multimodal Knowledge Agent (MKA) benchmark, used to evaluate AI agents' ability to handle visually complex files and complete real-world knowledge work tasks.
Google said its newly released Gemini 4 Carbon achieved second place among the tested models in this benchmark, with an average cost per completed task only about one-sixth that of the first-place model.
Google DeepMind senior product manager Rohan Doshi and principal engineer Ben Caine explained that many existing knowledge work benchmarks can be completed with text alone, while real-world work often involves a large amount of visual information. For example, architects need to cross-check architectural blueprints, engineers need to analyze engineering schematics, and professionals also need to organize information from different sources into presentations, reports, and spreadsheets.
The MKA benchmark is designed to evaluate AI agents' performance in these complex work scenarios. Google collaborated with domain experts from 10 industries and 83 occupations to build an evaluation set containing 200 tasks, covering business, architecture, education, engineering, finance, healthcare, legal and government, logistics and mapping, science, and software and data.
Each task simulates a multi-step professional project, including realistic work prompts and a set of multimodal input files. Google said these tasks would typically take experienced professionals 3 to 8 hours to complete. IT之家 noted that each task is also accompanied by expert-written scoring rubrics, containing on average about 70 yes/no checklist items, of which factual accuracy accounts for 75% and visual design accounts for 25%.
In terms of the evaluation mechanism, all tested models run on the same open-source agent framework Stirrup, using the same tools, sandbox environment, and interaction turn budget. The final deliverables are scored by an agent-based review system built on non-Google models, which opens all files, renders pages, checks spreadsheet formulas, and verifies results against the expert scoring rubrics. Google said that when domain experts manually scored the model outputs, their agreement rate with the automated review system's results reached 96%.
To prevent test questions from leaking into model training data, Artificial Analysis kept the 200 formal evaluation tasks private. At the same time, the project also published 10 additional tasks on Hugging Face for developers to understand the test format and evaluate their own agents.
Google's published test results show that Gemini 4 Carbon took second place in quality ranking among the tested models, but its average task cost is only one-sixth of the first-place model's. Google also noted that there are significant differences in efficiency among models completing the same batch of tasks, with the average number of interaction turns consumed per task ranging from 30 to 130 turns.
Currently, the MKA benchmark is available through Artificial Analysis, and developers can use the platform to evaluate their own models. Gemini 4 Carbon is available through the Gemini API in Google AI Studio.
Related reading: