FARTBench Core R01

FARTBench Core R01

Results

Results revealed under a signed authorisation after every answer was locked blind. Four human reviewers are the reference; three AI judges are compared with them, not substituted for them. This page reports every quantity on its own, in a fixed order; the leaderboard ranks configurations by their factual-only score (Overall Ranking V1).

Round
R01
Status
Revealed, frozen
Schema
fartbench-core-results-v2

Revealed, frozen

R01 fartbench-core-results-v2 2026-09-18T20:45:00Z

Frozen as fartbench-core-r01-results-v2-revealed at the reveal. Every value is carried unchanged from the blind analysis locked before the reveal; any revision will be a new, versioned document.

Machine-readable: RESULTS_V2.json (fartbench-core-results-v2, SHA-256 7ba59f77305b…)

What stood out

Descriptive highlights drawn from the results below, in no order of merit. Each is one dimension on its own; none is a ranking or an overall judgement.

  • Seedance 2.5 and Vidu Omni Pro 3.2a achieved human-consensus event success and primary source/cause success in all 4 scenes.
  • MiniMax H3 768p local showed the requested event in 4 of 4 scenes, versus 1 of 4 for MiniMax H3 2K API (same family, MiniMax H3).
  • LTX-2.5 Pro and Grok Imagine Video 1.5 showed the requested event in 0 of 4 scenes; the reviewers agreed it was absent each time.
  • The AI judges were least reliable on source and cause: they matched the human consensus on 56%–71% of the scenes' primary source/cause questions, and the three-judge panel met the pre-registered threshold on 0 of 4 of them, against 4 of 8 other factual questions (up to 100% agreement). The AI results are calibration evidence and do not replace the human reference.
  • FB-074 (Chair Confound) had the lowest human agreement of the 4 scenes — a 3-of-4 consensus on 64% of its questions, α 0.41 — so its configuration-specific results carry an uncertainty caveat (marked †).

Results by configuration

Each card shows, in order: whether the scene's event happened, whether the source or cause was right, every scene's facts with the raw votes, the three ratings with how many reviewers gave a number, reviewer agreement, refusal coverage, and how the AI judges fared on that configuration's clips. Cards are listed by name in a fixed order; the ranked view is the leaderboard.

Wan 3

Alibaba Wan 3 Model page →

Event happened
4 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Right source / cause
2 of 4 scenes · 2 without consensus
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: No consensus†
  • FB-512: No consensus
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did it clearly originate from the woman in the red jacket? Yes 4 → consensus Yes
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 4 → consensus Yes

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 4 → consensus Yes
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 1 · No 2 · Unclear 1 → no consensus
  • Was a separate fart-like event also present later? Yes 4 → consensus Yes
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Yes 2 · N/A (a sound absent) 2 → no consensus

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? No 4 → consensus No
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 2 · No 2 → no consensus
Sound realism
3.2 / 5 · n=16 · 0 N/A
Timing
3.4 / 5 · n=16 · 0 N/A
People and movement
3.6 / 5 · n=16 · 0 N/A
Reviewers
4 per clip · consensus on 9 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 7 of 9
  • gemini-3.7-flash 8 of 9
  • gemini-3.6-flash 8 of 9
  • Panel (2 of 3) 8 of 9
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

FLUX 3 Draft

Black Forest Labs FLUX 3 Model page →

Event happened
4 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Right source / cause
2 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: No†
  • FB-512: No
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 3 · No 1 → consensus Yes
  • Did it clearly originate from the woman in the red jacket? Yes 3 · Unclear 1 → consensus Yes
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 4 → consensus Yes

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 4 → consensus Yes
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? No 3 · Unclear 1 → consensus No
  • Was a separate fart-like event also present later? Yes 4 → consensus Yes
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Unclear 1 · N/A (a sound absent) 3 → consensus N/A (a sound absent)

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? Yes 3 · N/A (not observable) 1 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 1 · Unclear 3 → consensus Unclear
Sound realism
2.4 / 5 · n=15 · 1 N/A
Timing
3.1 / 5 · n=15 · 1 N/A
People and movement
3.1 / 5 · n=16 · 0 N/A
Reviewers
4 per clip · consensus on 12 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 11 of 12
  • gemini-3.7-flash 11 of 12
  • gemini-3.6-flash 8 of 12
  • Panel (2 of 3) 11 of 12
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

FLUX 3 Full

Black Forest Labs FLUX 3 Model page →

Event happened
4 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Right source / cause
1 of 4 scenes · 2 without consensus
  • FB-C02: No
  • FB-014: No consensus
  • FB-074: No consensus†
  • FB-512: Yes
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did it clearly originate from the woman in the red jacket? Yes 1 · Unclear 3 → consensus Unclear
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 1 · No 2 · Unclear 1 → no consensus

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 3 · No 1 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 2 · Unclear 1 · N/A (event absent) 1 → no consensus
  • Did the body and fitted clothing remain physically plausible through the event? Yes 3 · N/A (not observable) 1 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 2 · No 2 → no consensus
  • Was a separate fart-like event also present later? Yes 4 → consensus Yes
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Yes 2 · N/A (a sound absent) 2 → no consensus

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? Yes 1 · No 1 · Unclear 2 → no consensus
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 3 · Unclear 1 → consensus Yes
Sound realism
3.1 / 5 · n=15 · 1 N/A
Timing
3.1 / 5 · n=15 · 1 N/A
People and movement
3.1 / 5 · n=16 · 0 N/A
Reviewers
4 per clip · consensus on 7 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 5 of 7
  • gemini-3.7-flash 5 of 7
  • gemini-3.6-flash 6 of 7
  • Panel (2 of 3) 5 of 7
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Seedance 2.5

ByteDance Seedance Model page →

Event happened
4 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Right source / cause
4 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did it clearly originate from the woman in the red jacket? Yes 4 → consensus Yes
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 4 → consensus Yes

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 4 → consensus Yes
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 4 → consensus Yes
  • Was a separate fart-like event also present later? Yes 4 → consensus Yes
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Yes 4 → consensus Yes

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? Yes 4 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 4 → consensus Yes
Sound realism
3.9 / 5 · n=16 · 0 N/A
Timing
4.2 / 5 · n=16 · 0 N/A
People and movement
3.8 / 5 · n=16 · 0 N/A
Reviewers
4 per clip · consensus on 12 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 12 of 12
  • gemini-3.7-flash 12 of 12
  • gemini-3.6-flash 10 of 12
  • Panel (2 of 3) 12 of 12
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Veo 3.1 Fast

Google Veo 3.1 Model page →

Event happened
1 of 4 scenes · 1 without consensus
  • FB-C02: No
  • FB-014: Yes
  • FB-074: No consensus†
  • FB-512: No
Right source / cause
0 of 4 scenes · 3 without consensus
  • FB-C02: No consensus
  • FB-014: No
  • FB-074: No consensus†
  • FB-512: No consensus
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 1 · No 3 → consensus No
  • Did it clearly originate from the woman in the red jacket? Yes 2 · Unclear 1 · N/A (event absent) 1 → no consensus
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 3 · Unclear 1 → consensus Yes

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 3 · No 1 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? No 4 → consensus No
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 1 · Unclear 3 → consensus Unclear
  • Was a separate fart-like event also present later? Yes 2 · No 1 · Unclear 1 → no consensus
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? No 2 · Unclear 1 · N/A (a sound absent) 1 → no consensus

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 1 · No 3 → consensus No
  • Did the two people briefly react to it? Yes 1 · Unclear 1 · N/A (not observable) 2 → no consensus
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 1 · No 1 · Unclear 1 · N/A (event absent) 1 → no consensus
Sound realism
2.3 / 5 · n=16 · 0 N/A
Timing
1.7 / 5 · n=14 · 2 N/A
People and movement
2.4 / 5 · n=16 · 0 N/A
Reviewers
4 per clip · consensus on 7 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 3 of 7
  • gemini-3.7-flash 4 of 7
  • gemini-3.6-flash 3 of 7
  • Panel (2 of 3) 4 of 7
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Veo 3.1 Standard

Google Veo 3.1 Model page →

Event happened
3 of 4 scenes · 1 without consensus
  • FB-C02: No consensus
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Right source / cause
1 of 4 scenes · 3 without consensus
  • FB-C02: No consensus
  • FB-014: Yes
  • FB-074: No consensus†
  • FB-512: No consensus
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 1 · No 1 · Unclear 2 → no consensus
  • Did it clearly originate from the woman in the red jacket? Yes 2 · Unclear 2 → no consensus
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 2 · No 1 · Unclear 1 → no consensus

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 3 · No 1 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 3 · No 1 → consensus Yes
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 1 · No 2 · Unclear 1 → no consensus
  • Was a separate fart-like event also present later? Yes 3 · No 1 → consensus Yes
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? No 1 · Unclear 2 · N/A (a sound absent) 1 → no consensus

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 3 · No 1 → consensus Yes
  • Did the two people briefly react to it? Yes 4 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 1 · No 2 · Unclear 1 → no consensus
Sound realism
3.2 / 5 · n=16 · 0 N/A
Timing
2.6 / 5 · n=16 · 0 N/A
People and movement
3.1 / 5 · n=16 · 0 N/A
Reviewers
4 per clip · consensus on 6 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 5 of 6
  • gemini-3.7-flash 5 of 6
  • gemini-3.6-flash 4 of 6
  • Panel (2 of 3) 5 of 6
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

HappyHorse 1.1

HappyHorse Model page →

Event happened
2 of 4 scenes · 1 without consensus
  • FB-C02: No
  • FB-014: No consensus
  • FB-074: Yes†
  • FB-512: Yes
Right source / cause
1 of 4 scenes · 1 without consensus
  • FB-C02: No
  • FB-014: No consensus
  • FB-074: No†
  • FB-512: Yes
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did it clearly originate from the woman in the red jacket? Yes 1 · N/A (event absent) 3 → consensus N/A (event absent)
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 1 · N/A (nothing to follow from) 2 · N/A (not observable) 1 → no consensus

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 1 · No 2 · Unclear 1 → no consensus
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 1 · Unclear 1 · N/A (event absent) 2 → no consensus
  • Did the body and fitted clothing remain physically plausible through the event? Yes 3 · N/A (not observable) 1 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 1 · No 3 → consensus No
  • Was a separate fart-like event also present later? Yes 4 → consensus Yes
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Yes 1 · N/A (a sound absent) 3 → consensus N/A (a sound absent)

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? Yes 4 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 4 → consensus Yes
Sound realism
1.6 / 5 · n=10 · 6 N/A
Timing
3.3 / 5 · n=11 · 5 N/A
People and movement
3.5 / 5 · n=12 · 4 N/A
Reviewers
4 per clip · consensus on 9 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 5 of 9
  • gemini-3.7-flash 5 of 9
  • gemini-3.6-flash 4 of 9
  • Panel (2 of 3) 5 of 9
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Kling VIDEO 3.0 Omni

Kling 3 Model page →

Event happened
1 of 3 scenes
  • FB-C02: No
  • FB-014: No
  • FB-074: refused
  • FB-512: Yes
Right source / cause
1 of 3 scenes
  • FB-C02: No
  • FB-014: No
  • FB-074: refused
  • FB-512: Yes
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did it clearly originate from the woman in the red jacket? Yes 1 · N/A (event absent) 3 → consensus N/A (event absent)
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 3 · N/A (nothing to follow from) 1 → consensus Yes

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did the event occur during, or plausibly connect to, the box-lifting action? N/A (event absent) 4 → consensus N/A (event absent)
  • Did the body and fitted clothing remain physically plausible through the event? Yes 3 · N/A (not observable) 1 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

Refused by the provider. No clip exists, so there is no answer and no rating.

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? Yes 4 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 4 → consensus Yes
Sound realism
3.0 / 5 · n=4 · 8 N/A
Timing
3.5 / 5 · n=6 · 6 N/A
People and movement
3.1 / 5 · n=9 · 3 N/A
Reviewers
4 per clip · consensus on 9 of 9 questions
Coverage
3 of 4 scenes generated 1 refused
AI agreement
  • gemini-3.8-flash 7 of 9
  • gemini-3.7-flash 6 of 9
  • gemini-3.6-flash 7 of 9
  • Panel (2 of 3) 7 of 9
questions where the judge matched the human consensus

A factual result needs at least 3 of the 4 reviewers giving the same answer.

LTX-2.5 Fast

LTX-2.5 Model page →

Event happened
1 of 4 scenes · 1 without consensus
  • FB-C02: No
  • FB-014: No
  • FB-074: No consensus†
  • FB-512: Yes
Right source / cause
0 of 4 scenes · 2 without consensus
  • FB-C02: No
  • FB-014: No
  • FB-074: No consensus†
  • FB-512: No consensus
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 1 · No 3 → consensus No
  • Did it clearly originate from the woman in the red jacket? Yes 1 · N/A (event absent) 3 → consensus N/A (event absent)
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 2 · N/A (nothing to follow from) 2 → no consensus

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did the event occur during, or plausibly connect to, the box-lifting action? N/A (event absent) 4 → consensus N/A (event absent)
  • Did the body and fitted clothing remain physically plausible through the event? Yes 2 · N/A (not observable) 2 → no consensus High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 2 · No 2 → no consensus
  • Was a separate fart-like event also present later? Yes 1 · No 2 · Unclear 1 → no consensus
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Yes 2 · N/A (a sound absent) 2 → no consensus

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 3 · No 1 → consensus Yes
  • Did the two people briefly react to it? Yes 2 · Unclear 1 · N/A (event absent) 1 → no consensus
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 2 · Unclear 1 · N/A (event absent) 1 → no consensus
Sound realism
1.0 / 5 · n=6 · 10 N/A
Timing
2.8 / 5 · n=6 · 10 N/A
People and movement
2.7 / 5 · n=10 · 6 N/A
Reviewers
4 per clip · consensus on 5 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 3 of 5
  • gemini-3.7-flash 1 of 5
  • gemini-3.6-flash 2 of 5
  • Panel (2 of 3) 3 of 5
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

LTX-2.5 Pro

LTX-2.5 Model page →

Event happened
0 of 4 scenes
  • FB-C02: No
  • FB-014: No
  • FB-074: No†
  • FB-512: No
Right source / cause
0 of 4 scenes
  • FB-C02: No
  • FB-014: No
  • FB-074: No†
  • FB-512: No
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did it clearly originate from the woman in the red jacket? Yes 1 · N/A (event absent) 3 → consensus N/A (event absent)
  • Did the other passengers subsequently direct their attention or reaction toward her? N/A (nothing to follow from) 4 → consensus N/A (nothing to follow from)

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 1 · No 3 → consensus No
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 1 · N/A (event absent) 3 → consensus N/A (event absent)
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 2 · No 1 · Unclear 1 → no consensus
  • Was a separate fart-like event also present later? Yes 1 · No 3 → consensus No
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Yes 1 · N/A (a sound absent) 3 → consensus N/A (a sound absent)

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did the two people briefly react to it? N/A (event absent) 4 → consensus N/A (event absent)
  • Did they naturally recover and return toward their prior interaction, without another emission? N/A (event absent) 4 → consensus N/A (event absent)
Sound realism
1.0 / 5 · n=2 · 14 N/A
Timing
3.5 / 5 · n=2 · 14 N/A
People and movement
2.9 / 5 · n=11 · 5 N/A
Reviewers
4 per clip · consensus on 11 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 4 of 11
  • gemini-3.7-flash 4 of 11
  • gemini-3.6-flash 10 of 11
  • Panel (2 of 3) 4 of 11
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

MiniMax H3 2K API

MiniMax H3 Model page →

Event happened
1 of 4 scenes · 1 without consensus
  • FB-C02: Yes
  • FB-014: No
  • FB-074: No†
  • FB-512: No consensus
Right source / cause
2 of 4 scenes
  • FB-C02: Yes
  • FB-014: No
  • FB-074: No†
  • FB-512: Yes
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did it clearly originate from the woman in the red jacket? Yes 4 → consensus Yes
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 4 → consensus Yes

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 1 · N/A (event absent) 3 → consensus N/A (event absent)
  • Did the body and fitted clothing remain physically plausible through the event? Yes 2 · N/A (not observable) 2 → no consensus High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? No 4 → consensus No
  • Was a separate fart-like event also present later? No 4 → consensus No
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? No 1 · N/A (a sound absent) 3 → consensus N/A (a sound absent)

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 2 · No 1 · Unclear 1 → no consensus
  • Did the two people briefly react to it? Yes 3 · N/A (event absent) 1 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 3 · N/A (event absent) 1 → consensus Yes
Sound realism
2.4 / 5 · n=7 · 9 N/A
Timing
2.9 / 5 · n=8 · 8 N/A
People and movement
3.0 / 5 · n=11 · 5 N/A
Reviewers
4 per clip · consensus on 10 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 5 of 10
  • gemini-3.7-flash 5 of 10
  • gemini-3.6-flash 5 of 10
  • Panel (2 of 3) 5 of 10
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

MiniMax H3 768p local

MiniMax H3 Model page →

Event happened
4 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Right source / cause
2 of 4 scenes · 1 without consensus
  • FB-C02: Yes
  • FB-014: No consensus
  • FB-074: No†
  • FB-512: Yes
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did it clearly originate from the woman in the red jacket? Yes 4 → consensus Yes
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 4 → consensus Yes

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 3 · No 1 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? No 2 · Unclear 1 · N/A (event absent) 1 → no consensus
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? No 3 · Unclear 1 → consensus No
  • Was a separate fart-like event also present later? Yes 4 → consensus Yes
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Yes 1 · N/A (a sound absent) 3 → consensus N/A (a sound absent)

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? Yes 3 · Unclear 1 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 4 → consensus Yes
Sound realism
2.7 / 5 · n=15 · 1 N/A
Timing
3.0 / 5 · n=15 · 1 N/A
People and movement
3.1 / 5 · n=16 · 0 N/A
Reviewers
4 per clip · consensus on 11 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 11 of 11
  • gemini-3.7-flash 9 of 11
  • gemini-3.6-flash 10 of 11
  • Panel (2 of 3) 10 of 11
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Sora 2

OpenAI Sora Model page →

Event happened
1 of 4 scenes · 2 without consensus
  • FB-C02: No consensus
  • FB-014: Yes
  • FB-074: No consensus†
  • FB-512: No
Right source / cause
1 of 4 scenes · 1 without consensus
  • FB-C02: No
  • FB-014: Yes
  • FB-074: No consensus†
  • FB-512: No
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 2 · No 1 · Unclear 1 → no consensus
  • Did it clearly originate from the woman in the red jacket? Unclear 3 · N/A (event absent) 1 → consensus Unclear
  • Did the other passengers subsequently direct their attention or reaction toward her? No 1 · Unclear 1 · N/A (nothing to follow from) 1 · N/A (not observable) 1 → no consensus

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 4 → consensus Yes
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? No 2 · Unclear 2 → no consensus
  • Was a separate fart-like event also present later? Yes 2 · Unclear 2 → no consensus
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? No 1 · Unclear 1 · N/A (a sound absent) 2 → no consensus

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did the two people briefly react to it? N/A (not observable) 2 · N/A (event absent) 2 → no consensus
  • Did they naturally recover and return toward their prior interaction, without another emission? N/A (event absent) 4 → consensus N/A (event absent)
Sound realism
2.3 / 5 · n=11 · 5 N/A
Timing
2.9 / 5 · n=11 · 5 N/A
People and movement
2.8 / 5 · n=11 · 5 N/A
Reviewers
4 per clip · consensus on 6 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 4 of 6
  • gemini-3.7-flash 3 of 6
  • gemini-3.6-flash 3 of 6
  • Panel (2 of 3) 3 of 6
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Sora 2 Pro

OpenAI Sora Model page →

Event happened
2 of 4 scenes · 1 without consensus
  • FB-C02: Yes
  • FB-014: No
  • FB-074: No consensus†
  • FB-512: Yes
Right source / cause
1 of 4 scenes
  • FB-C02: No
  • FB-014: No
  • FB-074: No†
  • FB-512: Yes
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did it clearly originate from the woman in the red jacket? Unclear 4 → consensus Unclear
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 1 · No 3 → consensus No

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did the event occur during, or plausibly connect to, the box-lifting action? N/A (event absent) 4 → consensus N/A (event absent)
  • Did the body and fitted clothing remain physically plausible through the event? Yes 2 · N/A (not observable) 2 → no consensus High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? No 4 → consensus No
  • Was a separate fart-like event also present later? Yes 2 · No 2 → no consensus
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? No 1 · N/A (a sound absent) 3 → consensus N/A (a sound absent)

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? Yes 4 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 4 → consensus Yes
Sound realism
3.1 / 5 · n=10 · 6 N/A
Timing
3.3 / 5 · n=10 · 6 N/A
People and movement
2.9 / 5 · n=12 · 4 N/A
Reviewers
4 per clip · consensus on 10 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 8 of 10
  • gemini-3.7-flash 5 of 10
  • gemini-3.6-flash 9 of 10
  • Panel (2 of 3) 8 of 10
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Vidu Omni Pro 3.2a

Vidu Model page →

Event happened
4 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Right source / cause
4 of 4 scenes
  • FB-C02: Yes
  • FB-014: Yes
  • FB-074: Yes†
  • FB-512: Yes
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did it clearly originate from the woman in the red jacket? Yes 4 → consensus Yes
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 4 → consensus Yes

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the event occur during, or plausibly connect to, the box-lifting action? Yes 4 → consensus Yes
  • Did the body and fitted clothing remain physically plausible through the event? Yes 4 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? Yes 4 → consensus Yes
  • Was a separate fart-like event also present later? Yes 4 → consensus Yes
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Yes 4 → consensus Yes

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? Yes 4 → consensus Yes
  • Did the two people briefly react to it? Yes 4 → consensus Yes
  • Did they naturally recover and return toward their prior interaction, without another emission? Yes 4 → consensus Yes
Sound realism
4.1 / 5 · n=16 · 0 N/A
Timing
4.3 / 5 · n=16 · 0 N/A
People and movement
4.0 / 5 · n=16 · 0 N/A
Reviewers
4 per clip · consensus on 12 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 9 of 12
  • gemini-3.7-flash 11 of 12
  • gemini-3.6-flash 11 of 12
  • Panel (2 of 3) 11 of 12
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Grok Imagine Video 1.5

xAI Grok Imagine Model page →

Event happened
0 of 4 scenes
  • FB-C02: No
  • FB-014: No
  • FB-074: No†
  • FB-512: No
Right source / cause
0 of 4 scenes
  • FB-C02: No
  • FB-014: No
  • FB-074: No†
  • FB-512: No
Scene-by-scene facts, with every vote

FB-C02 Multi-Agent Source Attribution

  • Was exactly one fart-like event clearly present? No 3 · Unclear 1 → consensus No
  • Did it clearly originate from the woman in the red jacket? Yes 1 · N/A (event absent) 3 → consensus N/A (event absent)
  • Did the other passengers subsequently direct their attention or reaction toward her? Yes 2 · N/A (nothing to follow from) 2 → no consensus

FB-014 Fabric Stress Test

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did the event occur during, or plausibly connect to, the box-lifting action? N/A (event absent) 4 → consensus N/A (event absent)
  • Did the body and fitted clothing remain physically plausible through the event? Yes 3 · N/A (not observable) 1 → consensus Yes High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

FB-074 Chair Confound

  • Was a distinct chair-creak event present? No 4 → consensus No
  • Was a separate fart-like event also present later? No 4 → consensus No
  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? Unclear 1 · N/A (a sound absent) 3 → consensus N/A (a sound absent)

FB-512 Recovery Test

  • Was exactly one fart-like event clearly present? No 4 → consensus No
  • Did the two people briefly react to it? N/A (not observable) 1 · N/A (event absent) 3 → consensus N/A (event absent)
  • Did they naturally recover and return toward their prior interaction, without another emission? N/A (event absent) 4 → consensus N/A (event absent)
Sound realism
1.0 / 5 · n=1 · 15 N/A
Timing
3.0 / 5 · n=1 · 15 N/A
People and movement
3.4 / 5 · n=9 · 7 N/A
Reviewers
4 per clip · consensus on 11 of 12 questions
Coverage
4 of 4 scenes generated
AI agreement
  • gemini-3.8-flash 3 of 11
  • gemini-3.7-flash 1 of 11
  • gemini-3.6-flash 9 of 11
  • Panel (2 of 3) 3 of 11
questions where the judge matched the human consensus

† FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

A factual result needs at least 3 of the 4 reviewers giving the same answer.

Did the event happen, and was the cause right?

A clip counts only when at least 3 of the 4 reviewers gave the same answer. Clips without such a consensus are counted, named and never dropped; refused cells are excluded here and shown below.

R.1 Event success, by configuration

clips with consensus Yes

Table
Event success by configuration
ConfigurationCountOfRateNote
Wan 344100%
FLUX 3 Draft44100%
FLUX 3 Full44100%
Seedance 2.544100%
Veo 3.1 Fast1425%1 no consensus
Veo 3.1 Standard3475%1 no consensus
HappyHorse 1.12450%1 no consensus
Kling VIDEO 3.0 Omni1333%
LTX-2.5 Fast1425%1 no consensus
LTX-2.5 Pro040%
MiniMax H3 2K API1425%1 no consensus
MiniMax H3 768p local44100%
Sora 21425%2 no consensus
Sora 2 Pro2450%1 no consensus
Vidu Omni Pro 3.2a44100%
Grok Imagine Video 1.5040%

The scene's primary event question has consensus Yes. 4 human reviewers; a result needs 3 of 4 giving the same answer.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

R.2 Right source / cause, by configuration

clips with consensus Yes

Table
Primary causal success by configuration
ConfigurationCountOfRateNote
Wan 32450%2 no consensus
FLUX 3 Draft2450%
FLUX 3 Full1425%2 no consensus
Seedance 2.544100%
Veo 3.1 Fast040%3 no consensus
Veo 3.1 Standard1425%3 no consensus
HappyHorse 1.11425%1 no consensus
Kling VIDEO 3.0 Omni1333%
LTX-2.5 Fast040%2 no consensus
LTX-2.5 Pro040%
MiniMax H3 2K API2450%
MiniMax H3 768p local2450%1 no consensus
Sora 21425%1 no consensus
Sora 2 Pro1425%
Vidu Omni Pro 3.2a44100%
Grok Imagine Video 1.5040%

The scene's primary causal question — who did it, what it followed, whether the two sounds were told apart, whether they recovered — has consensus Yes.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

How convincing was it?

Ratings are 1 to 5 in the words the reviewers read, pooled over every number they gave. Each figure says how many numbers it rests on and how many answers were N/A; N/A is never a zero, and a rating never offsets a failed factual question.

R.3 Sound realism: “How real did the fart sound?”

rating, 1–5 · n = numeric ratings

mean median one standard deviation either side

Table
Sound realism by configuration
ConfigurationMeanMedian SDRatings (n)N/AClips rated
Wan 3 3.193.000.81 1604
FLUX 3 Draft 2.402.000.95 1514
FLUX 3 Full 3.133.000.88 1514
Seedance 2.5 3.944.000.66 1604
Veo 3.1 Fast 2.312.501.04 1604
Veo 3.1 Standard 3.193.500.88 1604
HappyHorse 1.1 1.601.500.66 1063
Kling VIDEO 3.0 Omni 3.003.000.71 481
LTX-2.5 Fast 1.001.000.00 6103
LTX-2.5 Pro 1.001.000.00 2142
MiniMax H3 2K API 2.433.001.29 792
MiniMax H3 768p local 2.733.001.34 1514
Sora 2 2.272.001.14 1153
Sora 2 Pro 3.103.000.83 1063
Vidu Omni Pro 3.2a 4.064.000.24 1604
Grok Imagine Video 1.5 1.001.00— 1151

Mean of the numeric ratings with one standard deviation either side; n is how many numbers the figure rests on, N/A how many answers had nothing to rate.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

R.4 Timing: “Did the timing make sense?”

rating, 1–5 · n = numeric ratings

mean median one standard deviation either side

Table
Timing by configuration
ConfigurationMeanMedian SDRatings (n)N/AClips rated
Wan 3 3.383.000.86 1604
FLUX 3 Draft 3.073.000.85 1514
FLUX 3 Full 3.073.000.68 1514
Seedance 2.5 4.194.000.73 1604
Veo 3.1 Fast 1.711.500.80 1424
Veo 3.1 Standard 2.562.001.00 1604
HappyHorse 1.1 3.273.000.75 1154
Kling VIDEO 3.0 Omni 3.503.500.96 662
LTX-2.5 Fast 2.833.000.90 6103
LTX-2.5 Pro 3.503.500.50 2142
MiniMax H3 2K API 2.883.001.05 883
MiniMax H3 768p local 3.003.001.15 1514
Sora 2 2.913.000.90 1153
Sora 2 Pro 3.303.500.78 1063
Vidu Omni Pro 3.2a 4.254.000.66 1604
Grok Imagine Video 1.5 3.003.00— 1151

Mean of the numeric ratings with one standard deviation either side; n is how many numbers the figure rests on, N/A how many answers had nothing to rate.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

R.5 People and movement: “Did the people and movement feel natural?”

rating, 1–5 · n = numeric ratings

mean median one standard deviation either side

Table
People and movement by configuration
ConfigurationMeanMedian SDRatings (n)N/AClips rated
Wan 3 3.634.000.86 1604
FLUX 3 Draft 3.133.000.48 1604
FLUX 3 Full 3.133.000.60 1604
Seedance 2.5 3.754.000.90 1604
Veo 3.1 Fast 2.383.000.93 1604
Veo 3.1 Standard 3.063.000.56 1604
HappyHorse 1.1 3.503.500.50 1244
Kling VIDEO 3.0 Omni 3.113.000.74 933
LTX-2.5 Fast 2.703.000.64 1064
LTX-2.5 Pro 2.913.000.51 1154
MiniMax H3 2K API 3.003.000.60 1154
MiniMax H3 768p local 3.063.000.75 1604
Sora 2 2.823.000.83 1154
Sora 2 Pro 2.923.000.86 1244
Vidu Omni Pro 3.2a 4.004.000.79 1604
Grok Imagine Video 1.5 3.443.000.50 974

Mean of the numeric ratings with one standard deviation either side; n is how many numbers the figure rests on, N/A how many answers had nothing to rate.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

How far the reviewers agreed

  • FB-074 FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.
  • FB-014/body_and_clothing_plausible High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

R.6 Agreement on each factual question

Krippendorff's alpha, 95% interval · raw votes

Table
Krippendorff's alpha per question, with its 95% interval and the raw votes
QuestionAlpha95% intervalClipsConsensusUnanimousSplitVotes
FB-C02 — Was exactly one fart-like event clearly present? 0.58 0.30 to 0.81 16 14 10 6 Yes 36 · No 24 · Unclear 4
FB-C02 — Did it clearly originate from the woman in the red jacket? 0.45 0.23 to 0.61 16 14 6 10 Yes 33 · N/A (event absent) 17 · Unclear 14
FB-C02 — Did the other passengers subsequently direct their attention or reaction toward her? 0.32 0.10 to 0.54 16 10 7 9 Yes 39 · N/A (nothing to follow from) 12 · No 7 · Unclear 4 · N/A (not observable) 2
FB-014 — Was exactly one fart-like event clearly present? 0.60 0.30 to 0.84 16 15 10 6 Yes 34 · No 29 · Unclear 1
FB-014 — Did the event occur during, or plausibly connect to, the box-lifting action? 0.61 0.33 to 0.84 16 13 10 6 Yes 28 · N/A (event absent) 26 · No 7 · Unclear 3
FB-014 — Did the body and fitted clothing remain physically plausible through the event? 0.07 -0.08 to 0.21 16 13 9 7 Yes 54 · N/A (not observable) 10
FB-074 — Was a distinct chair-creak event present? 0.29 -0.02 to 0.53 15 9 5 10 No 32 · Yes 18 · Unclear 10
FB-074 — Was a separate fart-like event also present later? 0.47 0.14 to 0.75 15 11 9 6 Yes 39 · No 17 · Unclear 4
FB-074 — Were the chair creak and the fart perceptually distinguishable as two different causal sounds? 0.17 -0.07 to 0.36 15 9 2 13 N/A (a sound absent) 31 · Yes 17 · No 6 · Unclear 6
FB-512 — Was exactly one fart-like event clearly present? 0.66 0.27 to 0.93 16 15 12 4 Yes 45 · No 18 · Unclear 1
FB-512 — Did the two people briefly react to it? 0.48 0.20 to 0.71 16 12 8 8 Yes 37 · N/A (event absent) 11 · N/A (not observable) 6 · No 5 · Unclear 5
FB-512 — Did they naturally recover and return toward their prior interaction, without another emission? 0.50 0.22 to 0.75 16 12 9 7 Yes 37 · N/A (event absent) 15 · Unclear 7 · No 5

Nominal alpha over exact answers, N/A reasons distinct. Beside it: how often a 3-of-4 consensus existed, unanimous and split clips, and every vote. 1 is perfect agreement; 0 is no better than chance.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

R.7 Agreement on the ratings

Krippendorff's alpha, 95% interval

Table
Krippendorff's alpha per question, with its 95% interval and the raw votes
QuestionAlpha95% intervalClipsConsensusUnanimousSplitVotes
How real did the fart sound? 0.54 0.37 to 0.67 252 — — —
Did the timing make sense? 0.51 0.35 to 0.63 252 — — —
Did the people and movement feel natural? 0.34 0.13 to 0.50 252 — — —

Interval alpha with N/A treated as missing.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

Pairwise exact agreement between reviewers: H01-H02 68% · H01-H03 69% · H01-H04 70% · H02-H03 77% · H02-H04 67% · H03-H04 66%.

Refusal coverage

R.8 Refused cells, by configuration

cells

Table
Refused cells by configuration
ConfigurationCountOfRateNote
Wan 3040%
FLUX 3 Draft040%
FLUX 3 Full040%
Seedance 2.5040%
Veo 3.1 Fast040%
Veo 3.1 Standard040%
HappyHorse 1.1040%
Kling VIDEO 3.0 Omni1425%
LTX-2.5 Fast040%
LTX-2.5 Pro040%
MiniMax H3 2K API040%
MiniMax H3 768p local040%
Sora 2040%
Sora 2 Pro040%
Vidu Omni Pro 3.2a040%
Grok Imagine Video 1.5040%

A refusal of the exact frozen prompt is the cell's result. It is coverage, not quality, and nothing stands in for it.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

R.9 The refusals

1 of 64 cells were refused by the provider; 63 produced a clip for review.

ConfigurationSceneOutcomeProvider's moderation response, verbatimReworded-prompt clip
Kling VIDEO 3.0 Omni FB-074 Refused (provider moderation) — no clip, no answers, no ratings “Prompt blocked as it may violate TOS, scat detected in prompt” kept as supplementary evidence; excluded from scoring

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

Do the AI judges agree with the humans?

Factual agreement with the human consensus is the primary AI result. Calibration thresholds were fixed before any AI answer was compared. The AI judges are measured against the human reviewers; they do not replace them.

R.10 Human–AI calibration

exact agreement with the human consensus

gemini-3.8-flash

0 of 4 primary causal questions calibrated (≥ 75%)

6 of 8 other questions calibrated (≥ 70%)

Ratings: MAE criterion met on 3 of 3 · useful discrimination shown on 0 of 3

gemini-3.7-flash

0 of 4 primary causal questions calibrated (≥ 75%)

4 of 8 other questions calibrated (≥ 70%)

Ratings: MAE criterion met on 3 of 3 · useful discrimination shown on 0 of 3

gemini-3.6-flash

2 of 4 primary causal questions calibrated (≥ 75%)

5 of 8 other questions calibrated (≥ 70%)

Ratings: MAE criterion met on 3 of 3 · useful discrimination shown on 0 of 3

Panel (2 of 3)

0 of 4 primary causal questions calibrated (≥ 75%)

4 of 8 other questions calibrated (≥ 70%)

Factual agreement with the human consensus, by question
Question Human-consensus clips gemini-3.8-flashgemini-3.7-flashgemini-3.6-flash Panel
FB-C02 Was exactly one fart-like event clearly present? 14 8/14 57% 9/14 64% 10/14 71% 9/14 64%
FB-C02 Did it clearly originate from the woman in the red jacket? primary causal 14 8/14 57% 9/14 64% 8/14 57% 9/14 64%
FB-C02 Did the other passengers subsequently direct their attention or reaction toward her? 10 7/10 70% 7/10 70% 9/10 90% 8/10 80%
FB-014 Was exactly one fart-like event clearly present? 15 11/15 73% 8/15 53% 10/15 67% 9/15 60%
FB-014 Did the event occur during, or plausibly connect to, the box-lifting action? primary causal 13 8/13 62% 6/13 46% 9/13 69% 7/13 54%
FB-014 Did the body and fitted clothing remain physically plausible through the event? 13 13/13 100% 13/13 100% 13/13 100% 13/13 100%
FB-074 Was a distinct chair-creak event present? 9 5/9 56% 3/9 33% 6/9 67% 6/9 67%
FB-074 Was a separate fart-like event also present later? 11 8/11 73% 7/11 64% 8/11 73% 7/11 64%
FB-074 Were the chair creak and the fart perceptually distinguishable as two different causal sounds? primary causal 9 5/9 56% 4/9 44% 7/9 78% 6/9 67%
FB-512 Was exactly one fart-like event clearly present? 15 11/15 73% 11/15 73% 11/15 73% 12/15 80%
FB-512 Did the two people briefly react to it? 12 10/12 83% 10/12 83% 8/12 67% 10/12 83%
FB-512 Did they naturally recover and return toward their prior interaction, without another emission? primary causal 12 8/12 67% 8/12 67% 10/12 83% 8/12 67%

✓ calibrated · ✗ not calibrated (a primary causal question below 75% is “not calibrated for the primary causal question”) · n/e no human consensus to compare with. Questions without a human consensus enter no denominator. The panel answer is never treated as the human truth.

Ratings — secondary, and never a substitute for the factual result

AI ratings against the human mean, where at least 3 of 4 reviewers rated
Ratinggemini-3.8-flashgemini-3.7-flashgemini-3.6-flash
Sound realism MAE criterion met · useful discrimination not demonstrated MAE 0.7886 · bias 0.0691 · n=41 correlation undefined · answered 3 on 100% of clips · no variation MAE criterion met · useful discrimination not demonstrated MAE 0.7798 · bias 0.0774 · n=42 correlation undefined · answered 3 on 100% of clips · no variation MAE criterion met · useful discrimination not demonstrated MAE 0.719 · bias -0.081 · n=35 correlation undefined · answered 3 on 100% of clips · no variation
Timing MAE criterion met · useful discrimination not demonstrated MAE 0.6159 · bias -0.1159 · n=41 Spearman 0.4841 · answered 3 on 61% of clips MAE criterion met · useful discrimination not demonstrated MAE 0.7183 · bias 0.1587 · n=42 Spearman 0.2317 · answered 3 on 50% of clips MAE criterion met · useful discrimination not demonstrated MAE 0.7452 · bias -0.1405 · n=35 Spearman 0.2322 · answered 3 on 80% of clips
People and movement MAE criterion met · useful discrimination not demonstrated MAE 0.5183 · bias -0.195 · n=50 Spearman 0.2012 · answered 3 on 88% of clips MAE criterion met · useful discrimination not demonstrated MAE 0.5317 · bias -0.075 · n=50 Spearman 0.137 · answered 3 on 80% of clips MAE criterion met · useful discrimination not demonstrated MAE 0.525 · bias -0.135 · n=50 Spearman 0.1258 · answered 3 on 96% of clips

The MAE criterion (≤ 1) is reported exactly as pre-registered. Whether a judge showed useful discrimination is recorded beside it as interpretation, not as a threshold: mae_threshold_met is exactly the pre-registered MAE <= 1.0 criterion of decision 0017, as computed. discrimination_demonstrated is this owner-recorded interpretation, carried as descriptive evidence; it is not a threshold and changes no pre-registered result.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

Detailed results

Show the full evidence: every question on every clip, per-scene outcomes and reviewer coverage

R.11 Every question, every configuration

human 3-of-4 consensus

Configuration FB-C02FB-014FB-074FB-512
Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3 Q1 Q2 Q3
Wan 3
FLUX 3 Draft
FLUX 3 Full
Seedance 2.5
Veo 3.1 Fast
Veo 3.1 Standard
HappyHorse 1.1
Kling VIDEO 3.0 Omni
LTX-2.5 Fast
LTX-2.5 Pro
MiniMax H3 2K API
MiniMax H3 768p local
Sora 2
Sora 2 Pro
Vidu Omni Pro 3.2a
Grok Imagine Video 1.5

consensus Yes consensus No consensus Unclear consensus N/A (reason in the cell) no 3-of-4 consensus refused Q2 underlined: the scene's primary causal question

  1. FB-C02 Multi-Agent Source Attribution: Q1 Was exactly one fart-like event clearly present? Q2 Did it clearly originate from the woman in the red jacket? Q3 Did the other passengers subsequently direct their attention or reaction toward her?
  2. FB-014 Fabric Stress Test: Q1 Was exactly one fart-like event clearly present? Q2 Did the event occur during, or plausibly connect to, the box-lifting action? Q3 Did the body and fitted clothing remain physically plausible through the event?
  3. FB-074 Chair Confound: Q1 Was a distinct chair-creak event present? Q2 Was a separate fart-like event also present later? Q3 Were the chair creak and the fart perceptually distinguishable as two different causal sounds?
  4. FB-512 Recovery Test: Q1 Was exactly one fart-like event clearly present? Q2 Did the two people briefly react to it? Q3 Did they naturally recover and return toward their prior interaction, without another emission?

Each cell is the human consensus on one question for one clip; its raw votes are in the cell's title. The primary causal question is underlined.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

R.12 Per-scene outcomes

clips

FB-C02 Multi-Agent Source Attribution

16 clips

Event happened
8 yes · 6 no · 2 no consensus
Right source / cause
6 yes · 8 no · 2 no consensus
Both of those
6 yes · 9 no · 1 no consensus
Every question
6 yes · 9 no · 1 no consensus

FB-014 Fabric Stress Test

16 clips

Event happened
9 yes · 6 no · 1 no consensus
Right source / cause
6 yes · 7 no · 3 no consensus
Both of those
6 yes · 7 no · 3 no consensus
Every question
6 yes · 7 no · 3 no consensus

FB-074 Chair Confound

15 clips · 1 refused, excluded

FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.

Event happened
8 yes · 3 no · 4 no consensus
Right source / cause
2 yes · 7 no · 6 no consensus
Both of those
2 yes · 7 no · 6 no consensus
Every question
2 yes · 8 no · 5 no consensus

FB-512 Recovery Test

16 clips

Event happened
11 yes · 4 no · 1 no consensus
Right source / cause
8 yes · 4 no · 4 no consensus
Both of those
7 yes · 5 no · 4 no consensus
Every question
6 yes · 6 no · 4 no consensus

Across all configurations, each outcome as yes, no and no consensus; refused cells excluded.

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

R.13 Reviewer coverage and N/A counts

responses

Configuration Clips Reviewers / clip Questions with consensus Sound realismTimingPeople and movement
n N/A n N/A n N/A
Wan 3 4 4 9 / 12 16 0 16 0 16 0
FLUX 3 Draft 4 4 12 / 12 15 1 15 1 16 0
FLUX 3 Full 4 4 7 / 12 15 1 15 1 16 0
Seedance 2.5 4 4 12 / 12 16 0 16 0 16 0
Veo 3.1 Fast 4 4 7 / 12 16 0 14 2 16 0
Veo 3.1 Standard 4 4 6 / 12 16 0 16 0 16 0
HappyHorse 1.1 4 4 9 / 12 10 6 11 5 12 4
Kling VIDEO 3.0 Omni 3 4 9 / 9 4 8 6 6 9 3
LTX-2.5 Fast 4 4 5 / 12 6 10 6 10 10 6
LTX-2.5 Pro 4 4 11 / 12 2 14 2 14 11 5
MiniMax H3 2K API 4 4 10 / 12 7 9 8 8 11 5
MiniMax H3 768p local 4 4 11 / 12 15 1 15 1 16 0
Sora 2 4 4 6 / 12 11 5 11 5 11 5
Sora 2 Pro 4 4 10 / 12 10 6 10 6 12 4
Vidu Omni Pro 3.2a 4 4 12 / 12 16 0 16 0 16 0
Grok Imagine Video 1.5 4 4 11 / 12 1 15 1 15 9 7

Source FARTBench Core R01 · human fartbench-core-r01-human-v3 · AI fartbench-core-r01-ai-v4 revealed-preliminary

Definitions

response state
yes, no, unclear, or n/a:<reason>; N/A is never No or Unclear, and N/A reasons are never collapsed
human consensus
at least 3 of the 4 reviewers give the exact same state; otherwise there is no consensus
event success
the scene's primary event question has consensus Yes
primary causal success
the scene's primary causal question has consensus Yes
strict scene success
primary event consensus is Yes and primary causal consensus is Yes
full factual success
every question reaches consensus, each is Yes or N/A, and both primary questions are Yes
outcome values
success; not_success (a consensus exists and it is not Yes); no_consensus
ratings
raw 1-5 values; mean, median and population SD of the numeric ones; N/A counted, never zero
reliability
Krippendorff's alpha: nominal over exact states for factual questions, interval for ratings; 95% interval by resampling clips; raw vote counts always beside it
AI factual calibration
exact-state agreement with the human consensus, scored only where a consensus exists; >= 70% for ordinary questions, >= 75% for a scene's primary causal question (decision 0017)
AI rating criterion
MAE <= 1.0 against the human mean where at least 3 of 4 humans rated (decision 0017), reported separately from rating discrimination
no composite
no single score is computed; every quantity is reported on its own
  • R00 and Core R01 both show source and cause attribution as the important weakness of the AI judges; this may be discussed descriptively after the reveal.