This may be the most imaginative 'pivot' among China's smart vehicle players:
This little desk lamp is famous worldwide—it's the logo of Pixar, the Hollywood special effects company behind Toy StoryPixar's logo. Of the past 30 Academy Awards for Best Visual Effects, 24 films used Pixar's effects technology.
Of course, Pixar is also famous for its founder—Steve Jobs. It was his startup after being 'shown the door' by Apple in 1985.
Not long ago, Pixar officially released its latest 3D rendering tutorial:
teaching creators how to use Pixar's official tools to process real-world 3D Gaussian splatting data. In the description, Pixar specially thanked a Chinese company for providing the 3D scene scanning solution Kosmo.
And that company happens to be a name that cannot be bypassed in the smart vehicle race:Hesai Technology—the leading maker of automotive LiDAR and a strong driver of the popularization of the smart car 'seatbelt.'
This, this, this—on one side, traditional, hardcore, rumbling automakers; on the other, the relaxed, cool, bustling Hollywood entertainment industry. Isn't Hesai's business layout just too imaginative?
How did the LiDAR company Hesai break into Hollywood?
RenderManis Pixar's core rendering technology, with an algorithm history even older than the company itself. Steve Jobs saw the potential of computer graphics as an art form back then and placed a heavy bet on it.
Today, GPU-accelerated XPU architecture has been implemented, and support for3D Gaussian splatting (3DGS), a brand-new 3D data format, has been added to RenderMan,and what helps RenderMan 'capture' the real world is Hesai's Kosmo.：
Here's how Pixar describes the value of Kosmo:
Film and television special effects require editing at the pixel level; the richer the data and information, the finer the adjustments that can be made. Kosmo allows modelers to walk into any place,capturing every spatial element, precise geometry, and real lighting of it.。
For example, the lamp (Pixar Lamp) and ball (Pixar Ball) in front of Pixar's Steve Jobs building at headquarters—capturing the space with Kosmo takes only 10-20 minutes, and the resulting high-quality 3DGS data enters the working environment directly via the USD format:
Next, Pixar engineers can move, scale, adjust transparency, paint in stylized ways... in the directly output model space, turning the entire plaza into a painting with flowing brushstrokes:
Reality → 3DGS → rendering. According to Hesai's data, capturing a space of about 100 square meters by hand-held Kosmo scanning takes roughly 15-20 minutes:
△ The operation is extremely simple. Scanned spatial objects appear on the display without mesh coverage; the remaining covered parts are yet to be scanned.
So for the film and entertainment industry, Kosmo's first value is the rapid extraction of real-world environmental information, with 100% faithful reproduction.
One more note,The phenomenon-level Chinese 3A blockbuster Black Myth: Wukong was made by Game Science, which back in the day used Hesai's LiDAR to capture models of various ancient buildings., and this is also one of the driving factors behind the birth of Kosmo.
Rapid modeling is only one aspect; what Kosmo is more recognized for isThe ability to directly output 3D Gaussian Splatting format means that the collected data requires no further complex processing and can be directly edited and tuned with various AI or manual tools。
So, the essence of Kosmo is actuallya real 3D world data engine, with multiple sensors highly integrated and well calibrated, ready to use out of the box，directly outputting 3D digital models that can be used for model training, can be manipulated and edited, and whose spatial relationships and colors and shapes are all free of distortion。
So,breaking into the Hollywood film industry is essentially just an 'egg' laid along the way while Hesai climbs the Mount Everest of physical AI。
Kosmo'smore 'original' application scenario is serving as the most important data channel entry point for embodied intelligence and the underlying models of physical AI。
The materials, object edges, and colors of real spatial environments are all reproduced at cinematic quality:
Even the paintings hanging on the wall can be reproduced losslessly, with color blocks, text, and positional relationships without any deviation:
Beyond the handheld device, Kosmo's cloud-based model can directly perform 3D semantic segmentation, separating out the various elements in a scene:
Why do this? The answer is:
The data assets fromKosmo's completed modeling can be directly used to build robot training tasks—this is in fact the goal pursued byworld modelsThe difference is thatKosmo's data comes from the real world, with more precise physics and causality, and model behavior is easier to explain and attribute。
Besides target objects and material colors, Kosmo itself also providesspatial relationshipsas an even more critical element:
For robots to go from perception to cognition and actively plan routes and strategies, spatial relationships are an essential decision factor and model training data.
And such a data entry point of "factory-installed mass production" and "early fusion" simply did not exist before across the entire embodied intelligence and physical AI赛道.
Embodied intelligence players, including other teams with mapping and data collection needs, actually have strong demand, and what they need goes far beyond LiDAR data.
It is just that for a long time, the way everyone played was to purchase different sensor components themselves and hand-build their own data collection devices—non-standard, low in reliability, expensive, and impossible to iterate.
But all of these pain points fall squarely within the comfort zone of Hesai, the LiDAR "number one": precision optoelectronic devices, automotive-grade standards, and engineering mass production.
So of course Kosmo integrates a LiDAR, but it is far more than an extension of Hesai's automotive LiDAR business, and not a show of force to sell more LiDAR units (as the LiDAR "number one" shipping millions of units annually, Hesai has long been widely recognized)—rather, it issolving the "reinventing the wheel" problem that has troubled everyone in the embodied intelligence and physical AI field—
the industry's first dedicated data terminal that is "ready to scan out of the box, and ready to use once scanned"。
What makes Hesai's Kosmo different?
Kosmo consists of two ends: Kosmo One, held by users, is the collection end, while Kosmo World in the cloud is the reconstruction end.
Inside Kosmo One there are four types of sensors: a custom LiDAR, an ultra-high-definition imaging system with up to 8K resolution, a 1-inch large-format CMOS, and a high-precision IMU.
The signals output by these four types of sensors are inherently different: point clouds are a pile of sparse three-dimensional coordinates, images are two-dimensional pixel arrays, IMU data is high-frequency time series, and so on.
The so-called early fusion means that before these signals enter any reconstruction algorithm, they are aligned at the hardware level first.
This happens along two dimensions—temporally, Kosmo achieves microsecond-level synchronized acquisition: the four types of sensors share the same clock source and trigger mechanism, and every frame of data is strictly bound by timestamp.
Spatially, the extrinsic calibration between the LiDAR and the camera determines whether each three-dimensional point in the point cloud can be accurately projected onto the corresponding pixel in the image.
Kosmo One integrates the three sensors in the same device; extrinsic parameters are calibrated at the factory and maintained during use with IMU assistance. The depth measured by the LiDAR can be directly matched point by point with image pixels, without relying on feature matching in software to guess the relative positions between sensors.
The output of early fusion is a set of spatiotemporally aligned multimodal data frames.
In each frame, the LiDAR point cloud, the image RGB pixels, and the IMU pose trajectory are bound under the same timestamp, jointly describing a complete observation of space made by the device at that moment.
Device-level early fusion is in fact of the same origin as the pixel-level early fusion Hesai did on the Picasa SPAD-SoC, where an RGB color filter is integrated above the SPAD pixels, and a single pixel simultaneously performs photon counting and ToF timing, directly outputting a spatiotemporally aligned colored point cloud.
Kosmo One extends a similar approach from the chip to a multi-sensor system.
Afterwards, the multimodal frames are uploaded to the cloud and enter Kosmo Spatial Core, abbreviated KSC, where they are converted into a 3D Gaussian splatting representation.
3DGS is a three-dimensional scene representation method proposed at SIGGRAPH 2023. It builds scenes with a set of 3D Gaussian distributions, each Gaussian having its own position, shape, transparency, and view-dependent color.
During training, starting from the sparse point cloud generated by camera calibration, these Gaussians are adjusted through the difference between rendered images and real images, finally yielding a cluster of three-dimensional points capable of rendering photorealistic images in real time.
But the original 3DGS has a shortcoming: it relies only on images, and monocular images cannot determine the true scale of a scene; places with little texture, such as white walls and glass, are easily computed incorrectly, and the spatial distribution of the Gaussians also lacks physical constraints.
Over the past two years, academia has been focused on solving this problem, with the direction being to inject the LiDAR's absolute depth into the initialization and optimization of 3DGS.
Mapping this back to Kosmo, the whole pipeline becomes like this:
Kosmo One completes microsecond-level synchronized acquisition, the ISP performs multi-view color and brightness consistency processing, and the data is uploaded to KSC.
KSC then uses the LiDAR point cloud to initialize Gaussian positions, uses depth as a geometric constraint in the optimization, and injects RGB information into the Gaussians' color parameters through a differentiable rasterization photometric loss.
The three signal streams work together within the same optimization framework, ultimately outputting a 3DGS representation in which position, shape, and color converge simultaneously.
Meanwhile, on this basis, KSC also synchronously exports two homologous data types, mesh and depth, with a unified spatial coordinate system, eliminating the need for cross-format registration.
Ultimately, a single reconstruction synchronously outputs three directly usable data forms for robot training: 3DGS, mesh, and depth.
Finally, KSC is also equipped with AIGC completion capability, performing automatic completion and material recognition for unscanned blind spots, and through AI semantic segmentation, independently identifying and labeling objects in the scene, enabling static 3D models to be directly converted into AI-trainable structured data.
Top intelligent vehicle players cannot remain just intelligent vehicles.
Kosmo's features actually far surpass smart vehicles, and even embodied intelligence application scenarios.
Film, gaming, surveying and mapping, robotics, and even real estate also need Kosmo's capabilities—VR home tours。
△ Pixar's creative lead Dylan and Hesai CEO Li Yifan
In one sentence: if data is 'oil,' then Kosmo is essentially the fully integrated infrastructure for the entire oil-extraction-plus-refining process, letting all the 'car owners' (AI models, the application layer, data demanders) pull up, fuel up, and go—ready to use out of the box.
This is the breadth of its applications; on 'depth,' Hesai has poured deeper thinking into Kosmo.
From autonomous driving to embodied intelligence, the debate between data simulation and real-world data collection has never stopped, and since the world-model concept became popular this year, this 'prominent school' of debate feels like it's about to be settled:
If you have a world model, why bother with large-scale real data collection?？
Seen through this logic, Kosmo seems to run 'against the AI tide.'
But the comparison between GPT-6 Astra and Kosmo's reconstructed scene data actually already makes the point:
At least at the current stage, the 'world model' has no way to answer this question:
The stronger AI becomes, the more problems world models/robots can reason about and solve—that is, the more scenarios unlocked for physical AI endpoints—but it also means the demands on generalization are higher.
Wouldn't that require even more fresh real-world data as a foundation?
Spatial data collection products like Kosmo may exist for a long time.
In other words, Kosmo isn't replaced as AI gets stronger; rather, the stronger AI becomes, the more useful it is—'pro-AI.'
Back to the skepticism: Kosmo itself is the kick start for a world model;
As forthe fundamental debate about world models—is AI really 'guessing the world' or truly understanding it? Academia and industry argue endlessly—butthrough Kosmo, Hesai has laid bare a more fundamental question that traditional large-model players have always avoided and ignored:：
models, especially during the training stage,if the data source can be aligned with the real physical world,wouldn't many problems be readily solved? Wouldn't many debates become unnecessary?
At the very least, only after clarifying the chain of where model capabilities come from and how to attribute explanations can discussions of the essence of world models avoid being 'flowers in a mirror, the moon in water.'
Hesai, this lidar supplier that emerged from the automotive world, truly deserves to be fully revalued.
First, the capabilities honed in the smart-vehicle track—precision equipment manufacturing, chip design, engineering-scale mass production, and more—have become the capability foundation for the physical AI era. Besides Kosmo, there are also its self-developed robot drive modules: small, powerful, and reliable:
Hesai's role has also shifted from a lidar supplier merely serving automakers to a physical AI infrastructure supplier—consider, for example, thatKosmo's main revenue may come not from hardware sales but from tokens consumed to generate data assets by the Kosmo brain.。
Second, Hesai's founding team now has more than just Hesai, one 'infrastructure' company; there is also the full-stack robotics company Sharpa:
Sharpa discovers problems in real-world deployment scenarios and feeds back requirements; Hesai keeps iterating and upgrading, delivering better solutions to the entire embodied intelligence track—and of course also earns direct revenue from Sharpa—a small closed loop of its own driving the industry's big closed loop.
Finally, Hesai has long since broken out of being a pure hardware company.
Kosmo itself is a combined hardware-and-model solution, but behind the model there is something less noticed: a data asset platform, where users' scanned data, Hesai's own official test data, and more can all be open-sourced:
CEO Li Yifan casually 'revealed' the ambition: Hesai deliberately made Kosmo feel 'Apple-like'—top-tier hardware R&D, design, and manufacturing to support a platform ecosystem covering all industries.
The further physical AI advances, the greater the demand for fresh real-world data will only become.
Hesai has fought its way out of the automotive circle and now stands at the gateway of physical AI data.
Next, it must prove how wide and how deep the commercial landscape along this path from sensors to data assets can go.
One thing is certain: the endgame for top smart vehicle players will not stop at smart vehicles.