FARTBench Core R01

Alibaba Wan 3 · Official Core R01 · rank 3= (shared with 2 others)

Wan 3

6/8Factual Score
4/4event happened
2/4right source / cause · 2 no consensus
2/4every question right
3.2 Sound realism · n=16 · 0 N/A
3.4 Timing · n=16 · 0 N/A
3.6 People and movement · n=16 · 0 N/A

Factual Score: one point per scene where the reviewers agreed the event happened, a second where the source or cause was also right. Ratings (1–5) are secondary and never change the rank.

The four scenes, with every reviewer's answer

A result needs at least 3 of the 4 reviewers (H01–H04) giving exactly the same answer; otherwise it is shown as no consensus.

FB-C02

Multi-Agent Source Attribution

2 of 2 points Event: Yes Source/cause: Yes Every question: Yes

  • Was exactly one fart-like event clearly present? event

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Did it clearly originate from the woman in the red jacket? primary cause

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Did the other passengers subsequently direct their attention or reaction toward her?

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Sound realism (1–5)

    H01 4H02 3H03 4H04 4 → 3.8 / 5 · n=4 · 0 N/A

  • Timing (1–5)

    H01 4H02 4H03 5H04 4 → 4.3 / 5 · n=4 · 0 N/A

  • People and movement (1–5)

    H01 5H02 4H03 4H04 4 → 4.3 / 5 · n=4 · 0 N/A

FB-014

Fabric Stress Test

2 of 2 points Event: Yes Source/cause: Yes Every question: Yes

  • Was exactly one fart-like event clearly present? event

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Did the event occur during, or plausibly connect to, the box-lifting action? primary cause

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Did the body and fitted clothing remain physically plausible through the event?

    H01 YesH02 YesH03 YesH04 Yes → Yes

    High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

  • Sound realism (1–5)

    H01 4H02 4H03 2H04 4 → 3.5 / 5 · n=4 · 0 N/A

  • Timing (1–5)

    H01 3H02 4H03 3H04 4 → 3.5 / 5 · n=4 · 0 N/A

  • People and movement (1–5)

    H01 4H02 4H03 3H04 4 → 3.8 / 5 · n=4 · 0 N/A

FB-074 †

Chair Confound

1 of 2 points Event: Yes Source/cause: No consensus Every question: No consensus

  • Was a distinct chair-creak event present?

    H01 UnclearH02 NoH03 NoH04 Yes → no consensus

  • Was a separate fart-like event also present later? event

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? primary cause

    H01 YesH02 N/A (a sound absent)H03 N/A (a sound absent)H04 Yes → no consensus

  • Sound realism (1–5)

    H01 4H02 2H03 2H04 3 → 2.8 / 5 · n=4 · 0 N/A

  • Timing (1–5)

    H01 3H02 3H03 3H04 4 → 3.3 / 5 · n=4 · 0 N/A

  • People and movement (1–5)

    H01 4H02 3H03 4H04 4 → 3.8 / 5 · n=4 · 0 N/A

FB-512

Recovery Test

1 of 2 points Event: Yes Source/cause: No consensus Every question: No

  • Was exactly one fart-like event clearly present? event

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Did the two people briefly react to it?

    H01 NoH02 NoH03 NoH04 No → No

  • Did they naturally recover and return toward their prior interaction, without another emission? primary cause

    H01 NoH02 YesH03 NoH04 Yes → no consensus

  • Sound realism (1–5)

    H01 3H02 3H03 2H04 3 → 2.8 / 5 · n=4 · 0 N/A

  • Timing (1–5)

    H01 3H02 3H03 1H04 3 → 2.5 / 5 · n=4 · 0 N/A

  • People and movement (1–5)

    H01 3H02 4H03 3H04 1 → 2.8 / 5 · n=4 · 0 N/A

Reliability caveats

  • † FB-074 FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.
  • FB-014/body_and_clothing_plausible High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

AI judges on these clips

  • gemini-3.8-flash7 of 9
  • gemini-3.7-flash8 of 9
  • gemini-3.6-flash8 of 9
  • Panel (2 of 3)8 of 9

Factual questions where the judge matched the human consensus. The AI judges are compared with the human reviewers; they never replace them. See AI calibration.

Community reviews

Rolling public blind reviews appear here once the review service is running. They are a separate layer and never change this configuration's Official Core R01 result.