NeurIPS 2026
1 PVG, Nanyang Technological University2 AMAP, Alibaba Group
Most world models for future prediction prioritize the aesthetic quality of generated videos, but offer only limited geometric consistency over time, due to the lack of explicit 4D modeling. In this paper, we introduce Plan4D, a possible next milestone towards building a generative plannable 4D world model that predicts future RGB videos together with dense 4D geometry, including depth, cameras, point clouds, and scene flow, from an observed scene context and text-specified actions. Since training data with paired actions and dense 4D annotations is inherently scarce, Plan4D leverages the complementary priors of foundational video generators and 3D reconstructors. Specifically, we propose a model-stitching architecture that couples Depth Anything 3 with LongCat-Video and augments it with an auxiliary DPT head for scene-flow prediction. The resulting model allows us to plan and generate complex, long-horizon, and fine-grained actions in dynamic 3D scenes, while substantially improving both visual fidelity and geometric accuracy of the generated future videos. Extensive experiments on both synthetic and real-world datasets demonstrate that Plan4D achieves state-of-the-art performance in generative 4D modeling of environmental reactions.
Observed context and a planned action specify how the scene evolves.
RGB, depth, cameras, and scene flow are synchronized readouts of one generated world.
A frozen stitched decoder provides geometric supervision to the video generator.
Action-conditioned world generation
Everyday interactions, generated from an observed image and a text-specified action.
pour eggs, tomatoes and pepper from plate into pan
brush garlic presser
spread jam on bagel
knead dough
put rice onto plate
stir pasta
All 81 time steps are retained. Point clouds are uniformly subsampled to at most 16,000 points per frame for the web. Views are aligned to each sequence's first camera and independently scaled for display.
Hold the stabilizer steady. Walk across the grassy area. Maintain the stabilizer's position. Ensure the camera stays balanced.
Camera tracks a black robot tailing a white one forward.
After the blueberries are poured into the bowl, a person pours a white liquid—likely yogurt or milk—from a small container into the bowl, allowing the liquid to flow over the blueberries. Once the pouring stops, the container is moved away from the bowl and placed on the counter. The bowl is then held steady by another individual, whose hands support it as the person who poured the liquid reaches for a spoon. Finally, the spoon is placed into the bowl, resting among the blueberries and the newly added liquid.
pour eggs, tomatoes and pepper from plate into pan
brush garlic presser
put rice onto plate
Up to 200,000 uniformly sampled scene points per method. All camera-frustum and trajectory points are retained. Each cloud uses its first camera orientation and an independent display scale.
All 81 time steps are retained. Frame previews contain up to 16,000 sampled points. Aggregated previews contain up to 200,000 scene points and all camera-trajectory vertices. Display scales are normalized independently.
The framework
A latent-to-geometry connection joins a video generator with a dynamic 3D reconstruction model.
A DPT-style motion head predicts dense 3D displacements from frozen DA3 features.
A compact spatiotemporal projection maps video latents into the reconstruction backbone.
Teacher distillation and annotated geometry calibrate the stitched decoder.
Flow matching and geometric supervision shape a visually and spatially consistent rollout.
Reference
@inproceedings{zhang2026plan4d,
title = {Plan4D: Generative Plannable 4D Worlds},
author = {Zhang, Hangwei and Dang, Rujing and Lin, Ruimin and
Wu, Jiahong and Chu, Xiangxiang and Zheng, Chuanxia},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}