| title | Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation |
|---|---|
| authors | Amber Yijia Zheng, Lu Liu, Raymond A. Yeh, Xi Yin |
| date | 2026-07-21 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.18789 |
| upvotes | 3 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Text-to-video generation has advanced significantly over the past five years through scaling of model size, data, and compute. Unlike model architecture, training data is often underexplored. Real-world data curation is complex and non-trivial, involving clip selection from raw videos and captioning to create video-text pairs for learning text-to-video mappings. We study how data distribution and caption quality impact text-to-video models. To enable controlled experiments, we introduce Moving Alphabet, a procedural testbed that renders letters with varying fonts, colors, sizes, and positions, moving in different directions and speeds against a black background. This design allows precise control over data distribution and caption quality by corrupting ground-truth metadata. Our experiments yield three findings: a) a diverse and balanced distribution of video content and duration is critical for generalization; b) caption quality significantly affects both model performance and training efficiency, suggesting that text-to-video models are bounded by video understanding capabilities; and c) classifier-free guidance and fine-tuning on high-quality data provide partial recovery from models trained on corrupted captions, but cannot fully compensate for poor pre-training data. We believe these insights can inform the development of large-scale text-to-video models, and we advocate for greater attention to the science of pre-training data. |
cite
@misc{zheng2026moving,
title={Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation},
author={Amber Yijia Zheng and Lu Liu and Raymond A. Yeh and Xi Yin},
year={2026},
eprint={2607.18789},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.18789}
}