| title | Beyond Pixels: From Video Priors to 4D Worlds |
|---|---|
| authors | Zihao Liu, Xiaolong Shen, Zhenglin Zhou, Ruijie Quan, Yi Yang |
| date | 2026-08-11 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.10744 |
| upvotes | 74 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | 4D generation synthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share a variational autoencoder (VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce direct latent-to-4D generation and instantiate it as Latent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained 4D decoder and refining it through frame-wise and global spatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiple video diffusion transformers within the same VAE family. On Text4D-200 and I4D-200, Latent-to-4D surpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality. |
cite
@misc{liu2026beyond,
title={Beyond Pixels: From Video Priors to 4D Worlds},
author={Zihao Liu and Xiaolong Shen and Zhenglin Zhou and Ruijie Quan and Yi Yang},
year={2026},
eprint={2608.10744},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.10744}
}