Author丨Zheng Jiamei
Editor丨Cen Feng
Zhipu's internal GLM-Omni made its first appearance in a paper. This research was jointly completed by Tsinghua University and Zhipu AI, with Tang Jie as corresponding author. As soon as the model name appeared in the paper, people on social media linked it to GLM-5.5. Although the paper does not disclose the version correspondence, it presents GLM-Omni's technical foundation and a set of experimental results on unified visual understanding and image generation.Reference link:https://arxiv.org/pdf/2610.21459One set of figures is especially striking: after image generation training, GLM-Omni correctly drew a specified number on a jersey in 72% of cases in the number task, yet when researchers removed the images and asked about the jersey number in text, the accuracy was only 2%. Knowledge the model can already reproduce in an image is almost impossible to answer when asked as a question. This hits exactly at a technical bottleneck of unified multimodal models. In the past, image understanding was responsible for seeing, image generation for drawing. Now the two capabilities are gradually entering the same architecture, and as a model learns to generate objects, maps and text patterns, it also encounters large amounts of visual information not described in language. But how much of what is learned from images can actually be used by language reasoning is hard to know just from generation results. Zhipu's paper attempts to break this question apart along the process of knowledge entering the model and then being invoked.
01
KTP: Leaving the Answer in the Image
To confirm whether generation training taught the model new knowledge, one must first bypass its existing memory. Asking a model to generate a real team's jersey, even if the number is drawn correctly, may just be because it read it long ago in text materials. Lei Feng Wang The researchers therefore built a fictional world, MagicWorld, assigning colors, numbers and spatial relations to entities that never existed, so that the target knowledge enters the model as much as possible only from the images of this training. For example, a fictional jersey called Arvenbrock is assigned number 10. During training, the text prompt only asks to generate this jersey, and the number 10 appears only in the target image. After the model finishes training, the researchers directly ask it for Arvenbrock's jersey number. If it can answer 10, there is reason to believe it carried the entity-attribute associations from the image into its text answer. This dataset is not only jerseys. The team designed 7 task categories — color, quantity, shape, spatial arrangement, relative size, numbers and text — with 50 pieces of knowledge each, 350 in total, then created images of each entity with different backgrounds and viewpoints, and after filtering kept 8552 training samples. The training text was also checked to minimize interference from answer leakage and uneven attribute distribution. Different tasks also spread out knowledge types: shapes and colors usually belong to limited categories, spatial arrangement requires understanding positional relations between objects, while jersey numbers and book titles require memorizing specific character sequences. With this batch of controlled knowledge, the researchers tested two paths separately: one had the model regenerate the target image and then check the attributes in it; the other provided no image, asking only in text. The former measures whether the model can reproduce the association during generation; the latter measures whether it can directly convert the association into a language answer. This distinguishes whether the model never learned an attribute, or can already draw it but cannot invoke it from the text side. The three models participating in the experiment adopt different architectures. BAGEL places understanding and generation in relatively independent expert branches, SenseNova-U1 directly models image pixels, while GLM-Omni starts from the pretrained model GLM-4.1V-Thinking and connects a discrete image Tokenizer based on X-Omni. The study also added 10000 text Q&A items unrelated to the target knowledge, letting the models become familiar with the answer format without getting the answers in advance.02
Can Draw but Cannot Say: Where Is the Problem
The gap measured directly is more general than the jersey case. BAGEL's average visual accuracy is 31.43%, with text answers only 8.86%; GLM-Omni is 54.00% and 14.57% respectively, and SenseNova-U1 61.14% and 20.00%. SenseNova-U1's visual accuracy in the number task even reaches 88%, yet pure text answers are 0%. The training objective of image generation is to have the model reconstruct visual content from names and prompts. As long as the entity-attribute association helps restore the image, it can be reinforced through training, but text Q&A must also follow the language output path to find the corresponding attribute and turn it into characters or words. Shared parameters do not mean these paths are already interconnected. Some knowledge was not even learned stably through the generation path — for example, BAGEL and SenseNova-U1 both had 0% visual accuracy in the spatial arrangement task — so subsequent training needed to distinguish insufficient learning from retrieval difficulty. The team first adopted Textual Alignment, adding text Q&A for some entities to see whether the model could learn from a few examples how to read the same kind of visual attributes. The 50 entities per category were split into two groups: all received image generation training, among which 25 entities additionally received 10 text Q&A examples each, while the remaining 25 were given no text answers and reserved for testing. The model needed to apply the approach learned from the first group to the second, rather than directly memorizing the test answers. The results diverged considerably. BAGEL's average text accuracy rose from 8.86% to 15.43%, SenseNova-U1's from 20.00% to 28.00%, but GLM-Omni's fell from 14.57% to 12.57%. SenseNova-U1's shape task went from 20% to 68%, and relative size reached 100%, yet for all three models the accuracy on the number and book title tasks of test entities remained 0%. This difference relates to the attributes themselves. Shapes and sizes have transferable classification and comparison rules, so Q&A examples from other entities can help; numbers and book titles, however, need to correspond to character sequences unique to each entity, and learning the answer sentence pattern is far from enough. More crucially, image generation and text Q&A were still split into two kinds of training samples; the models learned drawing and answering separately, but never practiced going from visual content to a language answer within the same response. The research therefore changed how the two kinds of supervision were organized.03
Training Images and Text Together
Interleaved Image–Text Alignment puts image generation and text answering into the same training sequence. When encountering a jersey number question, the model first generates the corresponding jersey, then organizes the text answer based on the number in the image. The image reconstruction objective and the language supervision act on the same entity attribute, giving the training process a direct connection compared to the two task types practicing separately. Lei Feng Wang (Official Account: Lei Feng Wang) The evaluation conditions did not change: during training images and text could be interleaved, but testing still required direct text answers, with no temporary image generation. GLM-Omni's average text accuracy rose from 12.57% in the textual alignment stage to 26.86%, the shape task went from 12% to 64%, and spatial arrangement from 0% to 40%. BAGEL's and SenseNova-U1's averages also reached 17.14% and 28.57% respectively. However, some color and quantity tasks declined; interleaved training cannot yet cover every visual attribute. Did the improvement come from image supervision, or just from adding more text reasoning content? The team ran an ablation on GLM-Omni: keeping the same interleaved responses, text supervision and independent image generation training, they masked only the image loss in the interleaved samples. The average text accuracy immediately dropped from 26.86% to 13.71%. This shows the image reconstruction training signal did affect language invocation, rather than merely giving the model more text to read. As for which parameters the model uses to share this information, the experiments have not yet localized it. Even so, knowledge such as numbers and book titles remains difficult to state directly and stably. The researchers simply took another path: since the model is better at drawing the answer, could it draw first before answering? Visual Chain-of-Thought (Visual CoT) allows generating the target image during reasoning, then recognizing attributes from the image, and finally answering the text question. This time GLM-Omni's average accuracy rose from 26.86% to 41.14%, BAGEL's from 17.14% to 20.57%, and SenseNova-U1's from 28.57% to 34.86%. The number task improved most noticeably: the three models originally all had 0% accuracy on direct text answers, but with Visual CoT they reached 4%, 36% and 20% respectively. In one case in the paper, SenseNova-U1 gave the wrong 69 when directly answering a jersey number, but after generating the jersey image and reading it, it answered the correct 31. This brings new trade-offs. Generated images can serve as an intermediate carrier for knowledge retrieval, but require extra computation, and if the image content is wrong, the answer may be led astray. The book title task still showed no improvement under Visual CoT; generating and reading complex characters remains difficult. Also, since the experimental knowledge comes from a controlled fictional world, this route's effectiveness in real dynamic scenarios remains to be further tested.04
One More Hurdle for Unified Multimodality
GLM-Omni's appearance with the paper gives concrete technical details of Zhipu's internal exploration in unified visual understanding and generation. The rounds of experiments in the research bring out a problem long obscured by generation results: what a model learns through images cannot necessarily be used directly for language answers. Interleaved image-text training improved cross-modal connections for some knowledge, and Visual CoT lets the model re-read information via generated images when needed. For multimodal models continuing toward unified architectures, the next challenges, besides drawing well and answering correctly, include how to make knowledge accumulated across different tasks effectively interconnected. As for whether GLM-Omni will enter some subsequent product version, more public information is still needed.Hop on board and tour the highlights of global top AI conferences
Exclusive access to:
Expert talk slides
Full conference reports
Popular paper explanations
Interviews with rising academic stars
Scan the QR code above
or click 「Read Original」 to follow the channel.
This is an original Lei Feng Wang article; unauthorized reproduction is prohibited. For details seethe reprint notice。