FARTBench Core · Round 1
Core R01
Sixteen generative video configurations were each asked for the same four scenes. The resulting clips were reviewed blind: first the facts of each scene, then how convincing it was.
Where the round stands
| Stage | State | Detail |
|---|---|---|
| Generation cohort | Frozen | 16 configurations x 4 scenes = 64 cells, frozen by digest before review. |
| Primary human cohort | Frozen | Four reviewers, fixed before any aggregation. |
| Human reviews H01-H04 | Locked | Each reviewer saw all 63 blind clips; each pass validated and locked. |
| Four-reviewer blind aggregate | Locked | 3-of-4 consensus over exact answers, rules fixed before it ran. |
| Blind AI judging, three judges | Locked | The same questions, protocol frozen before any clip was sent. |
| Blind human-AI comparison | Locked | Calibration thresholds fixed before any AI answer was compared. |
| Clip-to-configuration mapping | Revealed | Opened once, under a signed owner authorisation, after every digest and tag matched its commitment. |
| Results identity reveal | Complete | One preliminary results document, joined to configurations without changing any blind answer. |
| Overall Ranking V1 | Frozen | Factual-only ranking adopted after the reveal (not preregistered); frozen before the site used it. |
| Community blind review | Pending | Tested end to end on staging (consent, Turnstile, blind review, aggregation); closed in production until the owner opens it. Community data is CC0 1.0; it never changes the Official results. |
| Public deployment | Live | Live at https://fartbench.org since 2026-09-19, verified against the staging-verified build: identical files, frozen digests, review closed, apex host with www and HTTP redirects. |
The four scenes
FB-C02
Multi-Agent Source Attribution
Multi-agent source attribution
Primary causal question: “Did it clearly originate from the woman in the red jacket?”
FB-014
Fabric Stress Test
Movement-coupled event and fabric
Primary causal question: “Did the event occur during, or plausibly connect to, the box-lifting action?”
FB-074
Chair Confound
Acoustic confound
Primary causal question: “Were the chair creak and the fart perceptually distinguishable as two different causal sounds?”
FB-512
Recovery Test
Recovery
Primary causal question: “Did they naturally recover and return toward their prior interaction, without another emission?”
How a clip is judged
Every reviewer answers six questions per clip: three factual questions specific to the scene, each yes, no or unclear, and then three ratings from 1 to 5 that are the same in every scene:
- How real did the fart sound?
- Did the timing make sense?
- Did the people and movement feel natural?
The factual questions come first because they carry the benchmark's causal question. A clip can sound excellent and still fail its scene. Event success, source and cause, each rating, reviewer agreement and refusals are all reported separately; the leaderboard ranks by a factual-only score (Overall Ranking V1, adopted after the reveal). See the methodology.