LeRobot’s Humanoid Stack Separates Intent From Balance

Humanoid robots create a control problem that robot arms can largely avoid: every useful action also has to preserve balance. A manipulator policy can often emit joint or end-effector targets at a modest rate, but a biped must continually react to shifts in weight, contact, and momentum. The newly documented Unitree G1 workflow in LeRobot addresses that mismatch by dividing responsibility between a high-level vision-language-action (VLA) policy and a fast whole-body controller.

The design is important beyond this particular robot. It offers a practical way to think about learned humanoid systems: let one component decide what motion should happen, while another enforces the rapid coordination needed to make that motion physically plausible.

What the release connects

The published workflow begins with whole-body demonstrations collected through VR teleoperation. Human reference motion is encoded by SONIC into a compact, quantized 64-dimensional representation. A fine-tuned π0.5 policy then learns to predict chunks of those latent motion tokens from an instruction, camera observations, and robot state. On the G1, SONIC decodes each token using recent proprioceptive history and produces joint targets for the robot’s low-level controller.

The accompanying example uses about 100 teleoperated can-moving episodes, totaling roughly 71 minutes at 50 frames per second. Each sample includes three camera streams, a 31-dimensional state covering 29 joints and two grippers, and a 66-dimensional action consisting of the motion latent plus two gripper commands. The team reports fine-tuning the base policy for 12,000 steps on four H100 GPUs. Hardware modifications, the dataset, policy weights, and supporting documentation are linked from the release.

Why the intermediate representation matters

Predicting raw joint targets would force the task policy to learn two very different jobs from the same relatively small demonstration set. It would need to recognize the can, interpret the instruction, and choose a useful trajectory. At the same time, it would have to reproduce the continuous coordination that keeps a 29-degree-of-freedom body upright. Those objectives operate at different time scales and have different failure modes.

A motion token provides an interface between them. The VLA can focus on task-level variation, while the controller contributes a prior learned from a much broader motion corpus. This resembles other layered engineering systems: a route planner does not regulate an engine’s fuel injection, and a game agent need not synthesize every animation frame. The abstraction is useful when it hides complexity without hiding the signals required for safe execution.

There is a tradeoff. A latent space limits the policy to movements that its encoder and decoder represent well. That can improve data efficiency and stability, but unusual contacts, payloads, or postures may fall outside the controller’s experience. The boundary also complicates diagnosis: a failed reach could originate in visual understanding, token prediction, latent decoding, or low-level tracking. Logging each layer is therefore more valuable than judging only the final success or failure.

A practical evaluation checklist

A compelling demonstration is only the first evaluation stage. Teams adapting this architecture should separate at least four questions.

First, test the controller independently. Feed reference motions or known tokens and measure whether execution remains stable across starting poses, floor conditions, and modest disturbances. This establishes the capabilities the higher-level policy can actually call upon.

Second, evaluate the learned policy without letting controller robustness conceal poor decisions. Inspect predicted tokens, timing, object-selection errors, and recovery after an interrupted action. A robot that stays upright while reaching for the wrong object is a controller success but a task-policy failure.

Third, measure the complete loop under distribution shift. Change object locations, lighting, backgrounds, and operator-independent initial poses. Report task completion alongside falls, emergency stops, timeouts, and human interventions. These outcomes have different practical costs and should not be collapsed into one headline rate.

Finally, audit latency. Camera capture, preprocessing, network inference, token buffering, decoding, and actuator communication each consume part of the control budget. Average latency alone can be misleading; rare stalls and jitter may matter more because the body continues moving while a command is late.

Simulation results need a narrow reading

The release also describes a separate depth-based ball-dodging experiment built by adapting SONIC with reinforcement learning. Its reported 79.1% dodge rate comes from a three-seed simulation evaluation of resolved throws. Blank depth produced 0%, while privileged simulator state reached 97.4%. Those numbers support the value of visual information in that simulation setup; they are not a measured real-robot success rate. A physical demonstration clip shows feasibility, but does not substitute for a controlled hardware benchmark.

That distinction is a useful standard for humanoid reporting. Simulation can isolate perception and policy choices cheaply, while hardware exposes delay, calibration drift, imperfect depth sensing, actuator limits, and consequences of falling. Both are informative when their claims remain clearly separated.

The broader opportunity

LeRobot’s contribution is less a single policy than a set of interoperable seams: teleoperation into datasets, datasets into policy training, policies into whole-body controllers, and the same robot interface into simulation and physical hardware. Those seams can let researchers replace one layer without rebuilding every other layer.

For builders, the immediate lesson is to choose interfaces deliberately. Record enough state to reproduce failures, preserve timing information, and benchmark components before combining them. A compact learned action space can make humanoid learning more tractable, but its real value depends on transparent boundaries, reproducible assets, and evaluations that reveal which layer earned each success.