Paper digest: video diffusion and world models
How AI video went from still images to Sora and to Open-Sora, trained for about US$200,000. What 'world simulator' means and how to judge tools.
In February 2024, OpenAI showed Sora, which could generate a minute of realistic video from a text prompt. It suggested such models could become "simulators of the physical world." A year later, an open-source team reported training a model of similar quality for about US$200,000.
For studios and marketers, AI video has moved from novelty to a working tool. Knowing where the technology came from helps separate real abilities from marketing claims.
The papers
| Paper | Authors | Year | Link |
|---|---|---|---|
| Scalable Diffusion Models with Transformers (DiT) | William Peebles and Saining Xie | 2022, ICCV 2023 | arXiv:2212.09748 |
| Video generation models as world simulators (Sora technical report) | Brooks, Peebles and others (OpenAI) | 2024 | openai.com |
| Open-Sora: Democratizing Efficient Video Production for All | Zangwei Zheng and others | 2024 | arXiv:2412.20404 |
| Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k | Zangwei Zheng and 32 co-authors | 2025 | arXiv:2503.09642 |
Why this research exists
Diffusion models create images by starting with random noise and gradually cleaning it up into a picture. Until 2022, most used a design called a U-Net.
Video is much harder than still images. Every frame has to match the last, objects have to stay the same as they move, and the camera has to move without the scene warping. The amount of computing needed also grows quickly with length and resolution.
How the research builds
DiT: Transformers for images
William Peebles and Saining Xie replaced the U-Net with a Transformer, the design behind chatbots, explained in our Attention Is All You Need digest. They cut compressed images into small patches and treated each patch like a word.
They found that bigger Transformers made consistently better images. Their largest model set a record score of 2.27 on a standard ImageNet image quality test (lower is better).
Sora: patches in space and time
Peebles then co-authored OpenAI's Sora report. Sora extends the same idea to video. It compresses video, cuts it into "spacetime patches" that cover a small area over a short stretch of time, and uses a diffusion Transformer to generate them.
Training on videos at their original sizes let Sora produce widescreen, vertical and square video. OpenAI reported it could keep people and objects consistent even when they left the frame, though not always.
The report gives no details about the model's size or training. It also lists clear failures: Sora doesn't model the physics of things like glass shattering, and long videos can drift or have objects appear from nowhere.
Open-Sora: an open version
With Sora's details kept private, Zangwei Zheng and colleagues built an open alternative and released the code and model weights. Its design splits attention in two. It looks within each frame, then across time at each spot, which is much cheaper than comparing every patch with every other.
Open-Sora can make videos up to 15 seconds long at up to 720p resolution, from text or a starting image.
Open-Sora 2.0: the cost question
The follow-up focused on cost. The team trained a model they describe as commercial-level for about US$200,000, through careful choice of training data, better compression and efficient training. In human ratings and on the VBench test, they report it is comparable to HunyuanVideo, a leading open model, and Runway's Gen-3 Alpha.
Results that moved
| Paper | Key result |
|---|---|
| DiT | Bigger Transformers gave steadily better images, with a record ImageNet score |
| Sora | Up to a minute of high-definition video, at many sizes and shapes |
| Open-Sora | Free, open video model up to 15 seconds at 720p |
| Open-Sora 2.0 | Near-commercial quality for about US$200,000 of training |
What "world simulator" means
In AI research, a world model usually means a system that can predict what happens next, so a robot can plan its actions. OpenAI uses "world simulator" more loosely. Sora's videos often look physically consistent, which suggests the model has learned some rules of the world from watching video.
That is different from being a reliable physics engine. A video can look convincing and still get basic physics wrong.
Limits
- Tests favour good looks. Most scores reward attractive, prompt-matching video, not physical accuracy or consistency over long clips.
- Detail is limited by compression, which is one reason AI video often struggles with readable text on signs and packaging.
- Generic tests rarely cover what Caribbean studios need, such as darker skin tones under mixed lighting, fast dance movement or local brand text.
- Comparing a company's best demo clips with an open model's test scores is rarely a fair comparison.
Who should care
Researchers and engineers now have an open, documented recipe for video generation, including its cost.
For Caribbean studios and agencies, these papers help you question vendor claims. When a tool says it is "Sora-class", ask what clip length and resolution it was tested at. Then test it on what you actually make: faces, movement, products with logos and local settings. For tools that combine video with speech and images, see our Omni-Diffusion digest.
Bottom line
AI video now rests on one main recipe: compress the video, cut it into patches and generate it with a diffusion Transformer. Open-Sora showed that recipe can be reproduced openly and cheaply. Watch for better tests of consistency and physics, since that is where today's models still fall short.