IT Home On November 7, local time on November 4, Apple’s research team submitted a paper titled “Perceptive Humanoid Motion from Simulation” to the preprint platform arXiv, revealing the HALO autonomous control system for humanoid robots.
The research team also made the data generation, model training, and deployment code available, covering aspects such as action generation, action tracking, task scheduling, as well as simulation and real robot operation.
During the deployment stage, the research team used the YuShu Technology G1 humanoid robot, with a dedicated 3D printed protective shell installed on its back. It also has a built-in Mac mini (M4 Pro chip, 64GB unified memory), and the Mac mini is responsible for performing real-time inference calculations for the computationally intensive Orchestrator (VLM) and Motion Generator (DiT) on board. The on-board chip in the G1 focuses on running the Tracker (RL policy network) at high frequencies.
During execution, HALO is controlled by three layers in coordination. The top layer is a task scheduling module based on the Visual Language Model (VLM), defaulting to Qwen3.6-35B-A3B, and the system can also choose to call cloud models via API; the middle action generation layer uses another Qwen3.5-2B model; the bottom layer is an action tracker trained through reinforcement learning and policy distillation, operating at a frequency of 50 times per second.
According to the paper, the research team conducted 360 individual task tests in 6 real environments that were not scanned or mapped in advance, with an average success rate of 91%.
For long sequence composite tasks, 10 tests were conducted in a single real scenario with 3 combinations. The average success rate was 70%. Among them, the most complex task—“Bending over to pass through obstacles — moving towards the target — pushing boxes”—achieved 5 successes out of 10 tests, with a success rate of 50%.
According to the introduction, the research team used 250 NVIDIA H100 GPUs and spent approximately 48 hours synthesizing the main training data. The action generator was trained through imitation learning using 64 B200 GPUs; the action tracker was trained via reinforcement learning and policy distillation using 4 H100 GPUs.
However, the simulation-generated data is not completely free from human intervention. The team stated that adding new physical tasks still requires manual prompt writing and the design of key frames for actions, and some actions use human video or motion capture materials.
The research team also pointed out that currently, robots mainly rely on visual perception and have difficulty directly detecting contact. When an object leaves the camera’s field of view or is obscured, the performance of tasks involving more contact decreases. Adding tactile perception will be one of the directions for future improvement.
IT Home provides relevant links:
Project homepage: https://apple-aiml-research.github.io/ml-halo/
GitHub repository: https://github.com/apple-aiml-research/ml-halo