Mundo Separates Physical Simulation From Visual Rendering

A useful world model has to answer two different questions: what should happen next, and what should that event look like? Mundo treats those as separate engineering problems. It reconstructs a particular location from images, represents important objects with an explicit 3D proxy, updates that proxy through code and a physics engine, then asks a video model to turn the resulting guide frames into realistic footage.

That division matters more than the spectacle of a generated camera move. It creates an inspectable layer between a user request and the final pixels. A developer can reason about object identity, position, camera motion, and collisions before the generative renderer adds appearance. At the same time, the proxy is deliberately approximate, so a convincing output should not be mistaken for a geometrically exact replay.

A hybrid world instead of one opaque predictor

Mundo starts with several views or a video of a place. Its reconstruction process estimates cameras and scene geometry, initializes object boxes, and uses visual reasoning plus numerical fitting to refine the proxy. A separate review step checks the result against the available views. The scene is then expressed as a program whose state includes objects, poses, and physical properties. MuJoCo handles most demonstrated simulations, while rendered proxy frames communicate motion and camera information to the video model.

This architecture assigns different jobs to representations that suit them. Code is good at explicit state changes: move an object, open a drawer, or advance a timeline. A simulator is good at enforcing a chosen physical model. A video generator is good at synthesizing texture, lighting, and details that a sparse proxy does not contain. None of those components alone has to solve the entire problem.

The cost is that errors can enter at every boundary. A missed object in reconstruction never receives a reliable simulated state. A badly sized box can produce misleading contact. An uncertain mass or friction value can make a plausible scene evolve incorrectly. Finally, the renderer can add or lose details even when the proxy trajectory is coherent. Inspectability helps locate an error; it does not eliminate it.

Why the proxy is the central design choice

Bounding boxes are a pragmatic interface. They are cheap to edit, easy to label, and sufficient for many questions about free space, approximate placement, and camera paths. They also make a scene legible to tools that need structured objects rather than pixels. For a navigation scenario, a box around a table can be more operationally useful than a beautiful but inaccessible mesh.

The same simplification becomes fragile when contact geometry determines the outcome. Consider evaluating a robot that must insert a plug, grasp a mug by its handle, or slide a thin object under a door. A rectangular volume cannot describe the relevant surface detail. In such cases, a team would need a more detailed asset or a task-specific collision model, and should verify that the visual object, collision object, and semantic label still refer to the same thing.

This suggests a practical rule: choose scene fidelity according to the decision being tested. Approximate boxes may support route planning and broad object relocation. Manipulation with tight tolerances requires finer geometry and stronger calibration. The final video is evidence that the renderer followed a conditioning signal, not evidence that every hidden physical parameter is correct.

A better evaluation checklist

A world reconstructed from sparse views contains both observations and guesses. Evaluations should keep those apart. First, hold out one or more input views and compare projected proxy objects against them. This tests whether refinement generalizes beyond the views used for fitting. Second, label regions that no camera observed. Changes in those regions should be reported as inferred content rather than reconstruction accuracy.

Next, evaluate state and rendering independently. For state, inspect trajectories, contacts, collisions, and task completion directly in the simulator. For rendering, ask whether the generated frames preserve identities, relative positions, and camera motion specified by the proxy. A photorealistic failure and an unattractive but physically faithful result are different failures, and combining them into one visual score hides the distinction.

Long sequences need additional checks. Mundo generates extended videos in overlapping chunks, carrying a fixed prefix from one chunk into the next. Reviewers should therefore inspect boundaries for abrupt changes, then test whether objects that leave the frame return with the same appearance and state. A smooth transition at every boundary does not guarantee global consistency.

For robotics, the most important comparison is against the real task distribution. Record which properties are measured, estimated, or chosen by an agent. Vary uncertain parameters over reasonable ranges and see whether a policy still succeeds. If success disappears after a small change in friction, mass, or geometry, the simulation is better treated as a scenario generator than as validation of real-world reliability.

Where Mundo is useful today

The project is well suited to rapid spatial previews, controllable camera exploration, and constructing illustrative scenarios in a specific environment. Its explicit intermediate state may also help researchers generate variations while retaining more control than an end-to-end text-to-video prompt offers. For robot development, it can contribute to real-to-sim workflows when the task tolerates approximate geometry and when outcomes are checked at the simulator level.

The published limitations define a much narrower operational envelope than the polished videos might imply. Reconstruction is measured in minutes, rendering is computationally heavy, small-object interactions can hallucinate, and there is no long-horizon memory mechanism. Those constraints rule out real-time control and make fine manipulation or extended dynamic scenes especially risky without extra validation.

Mundo's strongest idea is therefore architectural: keep a manipulable account of the world, use established simulation for explicit transitions, and reserve generative video for appearance. That separation gives developers places to test, intervene, and substitute better components. Its value will depend less on whether a clip looks convincing than on whether each layer remains faithful to the decision the simulated world is meant to support.