Core R01 · results revealed and frozen
FARTBench
Can AI video models make the right sound, from the right person, at the right moment? 16 video generators got the same four short scenes; people judged every clip blind — first whether the right thing happened, then how convincing it was.
- video configurations
- 16
- scenes
- 4
- blind clips
- 63
- reviewers per clip
- 4
- AI judges, compared
- 3
Core R01 leaderboard
How the ranking works. Each configuration was asked for four scenes. It earns one point for every scene where at least 3 of the 4 reviewers agreed the requested event happened, and a second point only where they also agreed the source or cause was right — a Factual Score out of 8. Ties go to more scenes with every factual question right; anything still equal shares its rank.
| Rank | Configuration | Factual scoreScore | Event | Right source / causeCause | Sound | Timing | People | Reviewers |
|---|---|---|---|---|---|---|---|---|
| 1= | Seedance 2.5 ByteDance Seedance | 8/8 | 4/4 | 4/4 | 3.9 n=16 | 4.2 n=16 | 3.8 n=16 | 4/clip |
| 1= | Vidu Omni Pro 3.2a Vidu | 8/8 | 4/4 | 4/4 | 4.1 n=16 | 4.3 n=16 | 4.0 n=16 | 4/clip |
| 3= | Wan 3 Alibaba Wan 3 | 6/8 | 4/4 | 2/4 | 3.2 n=16 | 3.4 n=16 | 3.6 n=16 | 4/clip |
| 3= | FLUX 3 Draft Black Forest Labs FLUX 3 | 6/8 | 4/4 | 2/4 | 2.4 n=15 | 3.1 n=15 | 3.1 n=16 | 4/clip |
| 3= | MiniMax H3 768p local MiniMax H3 | 6/8 | 4/4 | 2/4 | 2.7 n=15 | 3.0 n=15 | 3.1 n=16 | 4/clip |
| 6 | FLUX 3 Full Black Forest Labs FLUX 3 | 5/8 | 4/4 | 1/4 | 3.1 n=15 | 3.1 n=15 | 3.1 n=16 | 4/clip |
| 7 | Veo 3.1 Standard Google Veo 3.1 | 4/8 | 3/4 | 1/4 | 3.2 n=16 | 2.6 n=16 | 3.1 n=16 | 4/clip |
| 8= | HappyHorse 1.1 HappyHorse | 3/8 | 2/4 | 1/4 | 1.6 n=10 | 3.3 n=11 | 3.5 n=12 | 4/clip |
| 8= | Sora 2 Pro OpenAI Sora | 3/8 | 2/4 | 1/4 | 3.1 n=10 | 3.3 n=10 | 2.9 n=12 | 4/clip |
What stood out
- Seedance 2.5 and Vidu Omni Pro 3.2a achieved human-consensus event success and primary source/cause success in all 4 scenes.
- MiniMax H3 768p local showed the requested event in 4 of 4 scenes, versus 1 of 4 for MiniMax H3 2K API (same family, MiniMax H3).
- LTX-2.5 Pro and Grok Imagine Video 1.5 showed the requested event in 0 of 4 scenes; the reviewers agreed it was absent each time.
- The AI judges were least reliable on source and cause: they matched the human consensus on 56%–71% of the scenes' primary source/cause questions, and the three-judge panel met the pre-registered threshold on 0 of 4 of them, against 4 of 8 other factual questions (up to 100% agreement). The AI results are calibration evidence and do not replace the human reference.
- FB-074 (Chair Confound) had the lowest human agreement of the 4 scenes — a 3-of-4 consensus on 64% of its questions, α 0.41 — so its configuration-specific results carry an uncertainty caveat (marked †).
How FARTBench works
-
Models generate the same scenes
Every video generator receives the same four written scenes. A provider refusal is recorded as a refusal, never as a bad video.
-
People review clips blind
Four reviewers watch each clip with sound, without knowing which model made it, and lock their answers.
-
Facts first, then realism
Did the right thing happen, from the right person? Then, separately: how real did it sound, and did the timing and movement feel natural?
The four scenes
- FB-C02 Multi-Agent Source Attribution On a crowded bus — did it come from the right person?
- FB-014 Fabric Stress Test Lifting a heavy box — did it happen with the lift?
- FB-074 Chair Confound A creaky chair — could you tell the creak from the fart?
- FB-512 Recovery Test Two people talking — did they carry on afterwards?
Review blind clips yourself
Watch anonymous clips and answer the same six questions our reviewers did — one clip or fifty. You never see which model made a clip. Community answers form their own leaderboard and never change the official Core R01 results.