This page summarises the method. The technical report sets out the full protocol, with every rule and definition. Read the technical report →
Six questions per clip, facts first
Each clip gets three factual questions specific to its scene, then three ratings that are the same everywhere. The factual questions carry the benchmark's causal claim — who produced the sound, what it followed, whether it could be told apart from a creaking chair, whether the scene recovered — so they are asked first, and a clip can sound excellent and still fail them. The four scenes and their questions are on the scenes page.
The ratings are asked in plain words, each with a label for every point on the scale:
- How real did the fart sound? — Obviously fake (1) to Completely real (5)
- Did the timing make sense? — Way off (1) to Exactly right (5)
- Did the people and movement feel natural? — Very unnatural (1) to Completely natural (5)
Not applicable is not a low score
When the event does not happen at all, there is no sound to rate and no cause to attribute. Forcing a number there would record noise — and a forced 1 would be worse than noise, because it would look like a measured failure of quality. So each answer is either a real answer or an explicit N/A with an enumerated reason, and a frozen dependency map says exactly when each N/A is allowed:
- An event question is always answerable and never N/A.
- N/A that depends on an earlier answer needs a no there — never an unclear.
- A no never forces N/A: two events, or one that was unclear, still leave something to rate.
- N/A is never a number. It never enters a mean, and every average states how many N/A answers it left out.
Blind, and blind in the bytes
Every clip carries only an opaque ID. Container metadata that could name a generator is stripped, and every file is padded with zeros to one common size, so neither a file's properties nor its size tells a reviewer anything. Reviewers see the clip, the scene's title and the six questions — not the generation prompt, not the model, not anyone else's answers.
Several reviewers, kept separate
4 people — the primary cohort, fixed before any aggregation — reviewed every clip, each in their own balanced order. A clip's answer to a factual question counts only when at least 3 of the 4 gave the same answer. Every reviewer's answers are kept exactly as given. Agreement is reported three ways — raw votes per clip, unanimous versus split, and Krippendorff's alpha with an interval — because with a handful of reviewers any single agreement statistic is uncertain. See human review.
AI judges ask the same questions
Three AI judges (gemini-3.8-flash, gemini-3.7-flash, gemini-3.6-flash) answer the same six questions in the same words, under a protocol frozen before any clip was sent. The allowed N/A combinations are enforced by the judges' structured output, not just requested. Judges receive the same blind bytes the humans watch and nothing else: no human answer, no model identity, no generation prompt.
The comparison that matters is factual agreement with the humans, gate by gate — above all on source and cause. An earlier round found judges that agreed well with a human ranking while being systematically lenient about who produced the sound; agreement on ratings is therefore never reported without factual agreement beside it.
The thresholds were fixed before any comparison: a judge is calibrated for an ordinary question at 70% exact agreement with the human consensus and for a scene's primary causal question at 75%. For ratings, the pre-registered criterion is a mean absolute error of at most one point. That criterion can be met by a judge that answers the middle of the scale for every clip, so a rating result is always shown with whether the judge actually told clips apart — for example, “MAE criterion met · useful discrimination not demonstrated”. A rating result never changes a factual one.
What gets reported, separately
- Event success — the scene's event happened: at least 3 of 4 reviewers answered Yes.
- Correct source or cause — the scene's primary causal question: at least 3 of 4 answered Yes.
- Scene-specific factual success — every gate of every scene.
- Sound realism, timing, people and movement — mean, median and spread, with the number of ratings and of N/A answers.
- Reviewer agreement — per question.
- Refusals — as coverage, never as a quality score.
- Human–AI agreement — per gate.
- Overall Ranking V1 — the leaderboard's order: a Factual Score out of 8, one point per scene where the event happened and a second where the source or cause was also right; ties go to more scenes with every question right, then the rank is shared.
The pilot-era protocol (Core v0.1, seven weighted dimensions) remains documented on the original methodology page. Core R01 uses the reduced, versioned questionnaire described here.