FARTBench Core R01

Black Forest Labs FLUX 3 · Official Core R01 · rank 6

FLUX 3 Full

5/8Factual Score
4/4event happened
1/4right source / cause · 2 no consensus
0/4every question right
3.1 Sound realism · n=15 · 1 N/A
3.1 Timing · n=15 · 1 N/A
3.1 People and movement · n=16 · 0 N/A

Factual Score: one point per scene where the reviewers agreed the event happened, a second where the source or cause was also right. Ratings (1–5) are secondary and never change the rank.

The four scenes, with every reviewer's answer

A result needs at least 3 of the 4 reviewers (H01–H04) giving exactly the same answer; otherwise it is shown as no consensus.

FB-C02

Multi-Agent Source Attribution

1 of 2 points Event: Yes Source/cause: No Every question: No

  • Was exactly one fart-like event clearly present? event

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Did it clearly originate from the woman in the red jacket? primary cause

    H01 UnclearH02 UnclearH03 UnclearH04 Yes → Unclear

  • Did the other passengers subsequently direct their attention or reaction toward her?

    H01 NoH02 YesH03 UnclearH04 No → no consensus

  • Sound realism (1–5)

    H01 4H02 4H03 3H04 3 → 3.5 / 5 · n=4 · 0 N/A

  • Timing (1–5)

    H01 3H02 3H03 2H04 4 → 3.0 / 5 · n=4 · 0 N/A

  • People and movement (1–5)

    H01 3H02 3H03 3H04 2 → 2.8 / 5 · n=4 · 0 N/A

FB-014

Fabric Stress Test

1 of 2 points Event: Yes Source/cause: No consensus Every question: No consensus

  • Was exactly one fart-like event clearly present? event

    H01 YesH02 YesH03 YesH04 No → Yes

  • Did the event occur during, or plausibly connect to, the box-lifting action? primary cause

    H01 YesH02 YesH03 UnclearH04 N/A (event absent) → no consensus

  • Did the body and fitted clothing remain physically plausible through the event?

    H01 YesH02 YesH03 YesH04 N/A (not observable) → Yes

    High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

  • Sound realism (1–5)

    H01 2H02 4H03 1H04 N/A → 2.3 / 5 · n=3 · 1 N/A

  • Timing (1–5)

    H01 3H02 2H03 2H04 N/A → 2.3 / 5 · n=3 · 1 N/A

  • People and movement (1–5)

    H01 3H02 4H03 3H04 3 → 3.3 / 5 · n=4 · 0 N/A

FB-074 †

Chair Confound

1 of 2 points Event: Yes Source/cause: No consensus Every question: No consensus

  • Was a distinct chair-creak event present?

    H01 YesH02 NoH03 NoH04 Yes → no consensus

  • Was a separate fart-like event also present later? event

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Were the chair creak and the fart perceptually distinguishable as two different causal sounds? primary cause

    H01 YesH02 N/A (a sound absent)H03 N/A (a sound absent)H04 Yes → no consensus

  • Sound realism (1–5)

    H01 3H02 3H03 2H04 4 → 3.0 / 5 · n=4 · 0 N/A

  • Timing (1–5)

    H01 3H02 3H03 3H04 4 → 3.3 / 5 · n=4 · 0 N/A

  • People and movement (1–5)

    H01 3H02 3H03 2H04 4 → 3.0 / 5 · n=4 · 0 N/A

FB-512

Recovery Test

2 of 2 points Event: Yes Source/cause: Yes Every question: No consensus

  • Was exactly one fart-like event clearly present? event

    H01 YesH02 YesH03 YesH04 Yes → Yes

  • Did the two people briefly react to it?

    H01 UnclearH02 NoH03 UnclearH04 Yes → no consensus

  • Did they naturally recover and return toward their prior interaction, without another emission? primary cause

    H01 UnclearH02 YesH03 YesH04 Yes → Yes

  • Sound realism (1–5)

    H01 4H02 3H03 3H04 4 → 3.5 / 5 · n=4 · 0 N/A

  • Timing (1–5)

    H01 4H02 3H03 3H04 4 → 3.5 / 5 · n=4 · 0 N/A

  • People and movement (1–5)

    H01 4H02 3H03 3H04 4 → 3.5 / 5 · n=4 · 0 N/A

Reliability caveats

  • † FB-074 FB-074 has comparatively low human consensus and reliability; conclusions about any configuration in this scene carry an explicit uncertainty caveat.
  • FB-014/body_and_clothing_plausible High raw agreement but low Krippendorff alpha, because almost every answer is the same; read the raw vote proportions beside alpha rather than treating alpha alone as poor reviewer quality.

AI judges on these clips

  • gemini-3.8-flash5 of 7
  • gemini-3.7-flash5 of 7
  • gemini-3.6-flash6 of 7
  • Panel (2 of 3)5 of 7

Factual questions where the judge matched the human consensus. The AI judges are compared with the human reviewers; they never replace them. See AI calibration.

Community reviews

Rolling public blind reviews appear here once the review service is running. They are a separate layer and never change this configuration's Official Core R01 result.