About FARTBench

Why this exists

Contemporary video-generation evaluation is good at photorealism, motion smoothness, and prompt adherence in the abstract. It is much less good at the awkward composite case: a brief physical event, produced by one specific body in a scene containing several, audible at the right moment, with clothing and posture behaving plausibly and the surrounding people reacting in proportion and in the right direction.

Flatulence turns out to be an efficient way to require all of that at once in eight seconds. It is cheap to specify, unambiguous to observe, and unusually hard to fake — a model that gets the sound right but attributes it to the wrong person has failed in a way that is impossible to argue with.

What is actually here

Core R01, the current round. 4 scenes, each asked of 16 configurations under the same generation rules: 64 cells, of which 63 produced a clip for review and 1 is covered by a refusal. Every clip was reviewed blind by 4 people with a frozen six-question questionnaire, and by 3 AI judges asking the same questions. The roster, the questionnaire, the record schemas and every digest are on the data page.

From the pilot era, kept as a labelled record. The Core v0.1.0 protocol with its 8 canonical test specifications, a versioned prompt-adapter policy, a rendered prompt pack covering 25 commissioned configurations across 13 model families, a scoring rubric, a 15-code failure taxonomy, provenance schemas, and a blind evaluation instrument. They are in the Archive and, as downloads marked “Pilot”, on the data page.

The technical report is the origin document: the rationale and the full conceptual benchmark. Read the technical report →

Official Core R01 results. The first Core round is published on the results page as a frozen, versioned record: every quantity on its own, with its sample counts, reviewer agreement and the digests of everything it was built from, and a leaderboard ranked by a factual-only score (Overall Ranking V1, adopted after the reveal). The pilot-era scoring pages are kept in the archive as a labelled historical record and carry no official score.

Who runs it

FARTBench is maintained by Cale Frombach as independent research. There is no institutional affiliation, no funding relationship with any model provider, no sponsorship, and no commercial arrangement with any vendor listed in the model registry.

The technical report is self-published. It is not peer reviewed, has no DOI, and has not been accepted anywhere. It is described that way everywhere it appears.

Vendor neutrality

Models are listed because they are commissionable, not because anyone asked for them to be listed. No vendor has reviewed a prompt, a rubric, or a result before publication, and no vendor will. Refusals and provider warnings are recorded as operational metadata rather than worked around; where a provider's safety system makes a comparison unfair, that is reported as a limitation of the comparison.

On the joke

The premise is a joke. The rest is not, and the two are not in tension: a benchmark whose subject matter nobody has a commercial stake in is a benchmark nobody has a reason to game. The methodology on this site is the methodology that would be used for a benchmark about robot manipulation or surgical video, applied to a scenario chosen for being hard rather than for being dignified.

If a result eventually shows that a frontier model reliably produces a well-timed, correctly-attributed, acoustically plausible event while three bystanders react appropriately, that will be a genuine finding about audiovisual world modelling. It will also be very funny. Both can be true.

Contact and corrections

Corrections to the protocol, the specifications, the adapter renderings, or the model registry are welcome and are published as revisions rather than applied silently. Errors in provenance are treated as more serious than errors in presentation.