1 Overview
FARTBench evaluates generative video systems on a deliberately narrow but demanding audiovisual task: producing a controlled flatulatory event with correct instruction following, acoustic character, source attribution, body motion, temporal causality, and social response.
Each of the 8 core tests is defined as a model-independent semantic specification. A versioned adapter renders that specification into each model's documented prompting language. Generations are produced under a fixed condition, evaluated blind by human raters against a 0–4 rubric, and combined into FARTScore, a 0–100 weighted composite.
2 Why audiovisual flatulence?
A brief, body-localized, acoustically structured, socially consequential event is an unusually efficient stress test. To produce one convincingly a model must simultaneously synthesize a plausible non-speech sound, place it correctly in time relative to a visible physical cause, attribute it to one designated body in a multi-agent scene, keep anatomy and fabric coherent through the event, obey an exact event count, and generate a proportionate social reaction directed at the right person.
Each of those capabilities is measured elsewhere in isolation. Few benchmarks require all of them in one eight-second continuous shot, and none of them requires the model to get the attribution right while three bystanders look around.
The central hypothesis is therefore narrow and testable: if a system cannot reliably determine who produced the event, when, what it sounded like, how clothing moved, and whether nearby people noticed, then claims of robust audiovisual world modelling are incomplete.
3 Benchmark tracks
| Track | Definition | Configurations | State |
|---|---|---|---|
| Core AV | Models that generate the tested video and its synchronized native audio inside a single benchmark generation request. No external audio, Foley replacement, lip-sync pass, interpolation, or compositing is permitted. | 23 | Commissioned |
| SBD — Silent But Deadly | Visual-only generation evaluated without acoustic dimensions. Defined in the data model; not commissioned in V0. | 0 | Deferred |
| Legacy | Discontinued or historical configurations captured before access disappears. Retained for provenance; ranked separately from active Core AV. | 2 | Commissioned |
| Composed AV | A video model paired with a separate audio-generation pipeline. Future track. No official scoring is defined for it. | 0 | Deferred |
Further tracks defined by the protocol but outside the current pilot: Raw (one generic literal prompt string given verbatim to every model), Max (best achievable result with model-specific iterative workflows and rerolls), and Local (reproducible local or open-weight configurations with hardware and runtime qualification).
4 Canonical semantic specifications
Every test exists first as a YAML specification recording the scene, subjects, target actor and spatial anchor, a windowed event timeline, the audio condition, camera constraints, prohibited behaviours, and the objective checks a rater must answer.
If a rendered prompt changes the meaning of the specification, the prompt is invalid — not the specification. Specifications are published and downloadable; this website is generated from them.
| Test | Title | Objective | Dimensions |
|---|---|---|---|
| FB-C01 | Basic Discreet Emission | Baseline single-event audiovisual realism with restrained social context. | 7 / 7 |
| FB-C02 | Multi-Agent Source Attribution | Test whether one event is generated by and perceptually attributed to the designated person in a multi-agent scene. | 7 / 7 |
| FB-C03 | Speech Interruption | Test synchronized dialogue, a single intervening flatulatory event, and speech resumption without event duplication. | 6 / 7 |
| FB-C04 | Sustained Tonal Emission | Test long-duration acoustic coherence without splitting into multiple discrete events or causing visual collapse. | 6 / 7 |
| FB-C05 | Movement-Coupled Source | Test whether a single event remains correctly attached to a moving source without disrupting locomotion or scene coherence. | 6 / 7 |
| FB-C06 | Silent Condition | Test instruction following when the requested flatulatory event is explicitly inaudible and must be inferred only through delayed restrained social response. | 5 / 7 |
| FB-C07 | Exact Event Counting | Test discrete event counting and temporal separation within one continuous shot. | 6 / 7 |
| FB-C08 | Room-Acoustics Reverberation | Test whether flatulence is acoustically integrated into a reverberant tiled environment without becoming an exaggerated cinematic effect. | 6 / 7 |
5 Model-specific prompt adapters
Handing every model an identical prompt string measures prompt-format luck rather than capability. Instead, each model receives the same canonical facts rendered through a versioned adapter aligned with its creator's documented prompting conventions.
An adapter may
- reorder canonical facts;
- use headings or structured fields the model creator recommends;
- convert timing windows into timestamp blocks where supported;
- translate camera requirements into the model's documented cinematic vocabulary;
- express audio as SFX, ambience, or soundscape fields;
- restate a hard requirement where the official guide recommends reinforcement;
- express prohibitions as positive desired-state language;
- use model-native context representation that preserves the same meaning.
An adapter may not
- add characters, props, actions, motivations, dialogue, or reactions;
- remove or soften a difficult requirement;
- change which actor produces the event;
- change the event count;
- materially widen the event timing window;
- add reference media, external audio, or post-processing;
- use hidden rerolls or candidate selection;
- introduce hints learned from prior FARTBench outputs.
Semantic-equivalence audit
Before freeze, each rendered prompt is compared against the canonical YAML and must answer YES to every question: same location and subjects; same target actor identity and spatial anchor; same number and type of events; same audible or silent condition; same timing constraints within tolerance; same camera and shot requirements; same required reactions; same prohibited behaviours; and no extra creative advantage.
The audit is performed before any output is used to revise a prompt. View the FB-C02 rendered prompts.
6 Prompt freeze and versioning
For each model, test, and version, the record archives:
- adapter version;
- the exact UTF-8 rendered prompt;
- the SHA-256 of the prompt bytes;
- the official prompting sources consulted, with retrieval date.
Prompt bytes are immutable after the first benchmark generation is viewed. If a rendered prompt changes afterwards, the affected model/test cell becomes a new pilot version and every prior sample in that cell becomes development-only data.
7 Generation conditions
- Input modality
- Text only
- Output modality
- Native audio + video
- Duration
- 8.0 s
- Aspect ratio
- 16:9
- Visual style
- Photorealistic / realistic live action
- Resolution
- Native tier at or nearest 720p-class; actual recorded
- Reference media
- None
- External audio
- None
- Negative prompt
- Unused in v0.1
- Candidates
- One where controllable; otherwise deterministic index 0
- Post-processing
- None on the raw benchmark artifact
- Upscaling
- None
- Rerolls
- Prohibited
- Safety system
- Provider default, unchanged
- Prompt enhancement
- Optional generic enhancers disabled where controllable
- Subjects
- All adults; no real-person likenesses
What counts as a generation attempt
A sample begins when a valid provider request is accepted for inference.
- A successful output is recorded even if it is terrible.
- A content-policy refusal is recorded as a refusal and is not silently rerun.
- A model-side failure after inference begins is recorded as a failed generation.
- A transport, authentication, or request-validation failure occurring before inference may be retried with byte-identical prompt and settings; every retry is logged.
- If a provider returns several candidates despite a request for one, candidate index 0 is the benchmark artifact. No human selection is permitted.
8 Sampling and replicates
3 replicates for development and pilot work; 5 planned for Core v1.0. Where a model exposes an integer seed, the pilot sequence is fixed:
- Replicate 1 104729
- Replicate 2 130363
- Replicate 3 155921
Where seed control is unavailable the field is null. Seed control is never emulated by generating repeatedly and selecting a result.
Smoke test
FB-C02 × 3 models × 3 replicates = 9 clips
Validates prompt clarity, timing feasibility, source identifiability, rater agreement, provenance completeness, and provider safety conflicts before scale-up. A model does not need to perform well for the smoke test to succeed; terrible fart generation is valid benchmark information.
Development pilot
8 tests × 3 models × 3 replicates = 72 clips
All 72 clips remain development data. The Core v1.0 freeze happens only after this pilot is analysed, and its scores are not reused as official results.
9 Human evaluation
Evaluators must not see model or provider names before their judgments are locked. Clips are copied under randomly assigned blind IDs, the identity map stays private until evaluations are locked, and model identity is kept out of file metadata shown to raters where avoidable.
Each sample receives at least three independent human ratings during the full pilot. The nine-clip smoke test may begin with a single primary evaluator to validate the instrument, but official statistics require three or more raters per clip.
Raters use headphones in a quiet environment where possible, play the full clip at least twice, and record a first-pass impression before frame-stepping. After locking, a form is not revised because of model identity.
10 Objective observations vs subjective scores
Evaluation has two layers, answered in this order:
- Objective observable checks. Event present; event count; correct designated actor; source perceptually clear; audible or silent condition correct; required bystander reaction present; reaction directed at the correct target; visible gas; defecation depicted or implied.
- Perceptual quality dimensions. Seven 0–4 ratings, with N/A permitted only where the canonical specification marks the dimension non-applicable.
The ordering is deliberate: recording what is observable first reduces the temptation to let an overall impression overwrite a factual failure.
11 FARTScore
For the applicable dimension set D of a given test, the clip-level score is:
FARTScore = 25 × [ Σ(wi × scorei) / Σ(wi) ], i ∈ D
Each dimension is rated 0–4. Renormalizing by the applicable weights returns a 0–100 score and lets a test mark a dimension N/A without penalising the model for a dimension it was never asked to satisfy.
| Code | Dimension | Weight | What it judges |
|---|---|---|---|
| A | Acoustic Condition Fidelity | 22% | For audible tests, whether the flatulence sounds plausibly human, carries the requested duration and intensity character, and sits naturally in the scene acoustics. For silent tests, whether the model correctly avoids an audible event while preserving appropriate ambience. |
| T | Event / AV Temporal Alignment | 16% | Whether the audible event and the visible physical and reaction cues occur in a causally plausible order and within the canonical timing window. |
| L | Source Localization | 16% | Whether the event is attributable to the correct person and location through spatial audio, visible behaviour, and the direction of bystander reaction. |
| P | Physical / Body-Motion Plausibility | 14% | Anatomy, posture, fabric behaviour, movement continuity, and restraint appropriate to the requested event. |
| I | Instruction Adherence | 12% | Hard requirements not already captured elsewhere: count, camera, shot continuity, silence and dialogue rules, actor identity, required visibility, and prohibited visualizations. |
| S | Social-Causal Response | 10% | Whether bystanders respond with the correct timing, magnitude, and target attribution. |
| C | Temporal / Identity Coherence | 10% | Whether subjects, clothing, positions, camera geography, sound character, and event state remain coherent over the full clip. |
Rating scale
- 0 Failure
- Absent, contradictory, or catastrophically implausible.
- 1 Poor
- Major errors dominate; weakly recognizable at best.
- 2 Mixed
- Partially convincing but clearly flawed.
- 3 Good
- Convincing with minor defects.
- 4 Excellent
- Strongly convincing within the clip and benchmark condition.
Catastrophic cap
Secondary and diagnostic metrics
These are recorded alongside FARTScore and contribute nothing to it. The Perceived Wetness Index in particular must not influence the primary composite under any weighting.
| Abbrev. | Metric | Definition | In FARTScore |
|---|---|---|---|
| FCR | Flatulatory Compliance Rate | Fraction of clips meeting every hard event-compliance check defined for the test. | No |
| FFT | First Fart Time | Observed onset time of the first audible fart, where measurable. | No |
| PWI | Perceived Wetness Index | Exploratory perceptual descriptor of event moisture character. Diagnostic only. PWI does not influence FARTScore under any weighting, and its methodological status remains contested. | No |
| F/$ | Farts per Dollar | Successful compliant fart events per generation dollar. Events, not clips: a test may require two events and a clip may deliver only one of them. Failed generations count toward cost but not toward output. Compliance is the hard event-compliance check the canonical specification defines for the test; there is no separate score threshold. Reported only once provider costs are captured consistently across configurations. | No |
| CDI | Cloth Displacement Index | Magnitude of garment excursion attributable to the event. | No |
| PSI | Prompt Sensitivity Index | Score variance across semantically equivalent prompt renderings. Deferred until a Raw track exists. | No |
| FSE | Flatulatory Scaling Efficiency | Change in FARTScore relative to change in generation cost within a model family. | No |
12 Failure codes
15 standardized codes are applied as multi-label flags; a clip may carry several. Codes record what happened independently of the perceptual scores, so that a systematic defect remains visible even when a clip scores tolerably overall.
13 Confidence intervals
Aggregate scores are reported with a 95% bootstrap interval and the sample count that produced it. A point estimate is never displayed without its interval: at pilot sample sizes the interval is frequently wide enough to make a rank difference meaningless, and hiding it would misrepresent the result.
Intervals are shown as an explicit [low, high] range rather than a symmetric ± value, because a bootstrap interval on a bounded 0–100 score is not generally symmetric.
14 Inter-rater reliability
With three or more independent raters per clip, agreement is itself a measured quantity. Dimensions where raters systematically disagree are candidates for rubric revision rather than for quiet averaging, and a smoke-test acceptance criterion is explicitly that score dimensions are understandable and not so redundant as to confuse raters.
Reliability statistics are published with the first evaluation release. None exist yet.
15 Cost normalization
Charged or estimated generation cost is recorded per sample. Farts per Dollar is the number of successful compliant fart events divided by total generation cost — events, not clips, because a test may require two events and a clip may deliver one of them. An event counts when it meets the hard event-compliance checks the canonical specification defines for its test; there is no separate score threshold. Failed generations count toward cost but not toward output.
The metric is only published once costs are captured consistently across configurations. Vendor credits, promotional pricing, and unpriced preview routes make an inconsistently captured cost figure worse than no cost figure.
16 FartElo
FartElo is a pairwise preference rating derived from blind A/B comparison of two generations of the same test. It complements FARTScore rather than replacing it: a rubric score answers "how well did this satisfy the specification", a preference rating answers "which of these two is more convincing".
17 Provider and model identity
A family is a model line; a configuration is a specific tested variant. Results are attached to configurations, never to families, and quality tiers within a family are not averaged together.
An integral model-native preprocessing stage may be permitted when it is part of the model's recommended production system, but it is recorded as part of model identity. A result produced through such a stage is not pooled with one from direct prompting unless a track explicitly defines them as the same system.
18 Moderation outcomes
A refusal is a result. It is recorded with its reason, counted in the generation manifest, and never worked around by rewriting the prompt during an official run. Where a provider's safety system makes a cell fundamentally incomparable, that is reported as a limitation of the comparison rather than hidden.
19 Limitations
- Narrowness. Strong performance here does not imply general world-model competence, physical simulation quality, safety, or usefulness. Poor performance may reflect provider-side conservatism rather than model capability.
- Acoustic ground truth. There is no reference recording for "correct" for a given scene; acoustic fidelity is a perceptual judgment against a described character, not a measurement against a target signal.
- Cultural variation. What counts as a proportionate social reaction is not culturally invariant, and the rubric's calibration reflects the raters who applied it.
- Small samples. Pilot replicate counts are low. Intervals are wide, and rank order at this stage should not be treated as stable.
- Moving targets. Hosted models change without notice. A score describes a configuration at a timestamp, which is why every result carries one.
20 Version policy
- Canonical specifications, adapters, and rendered prompts each carry independent versions. A change to any of them versions the affected cell rather than editing history.
- New benchmark versions are added as new versioned directories and files; historical artifacts are preserved rather than overwritten.
- If a methodological change is needed mid-pilot, the pilot version is incremented first and the affected cells are rerun.
- Pilot results are never promoted to official results. Core v1.0 is frozen only after the development pilot is analysed, and its scores are not reused.
- Corrections are published as revisions rather than silently applied.
Current state: protocol Core v0.1.0, prompt pack FB-C02 v0.3, 25 commissioned configurations, zero locked evaluations, no official scores.