Core R01 · release surface
Data and reproducibility
Everything needed to reproduce the benchmark procedure is published, and so is the Core R01 results document the results pages are built from.
Licences
Two data layers, two licences. Each licence covers its own layer and nothing else.
| Layer | Covers | Licence |
|---|---|---|
| Official Core R01 benchmark data and instrument | RESULTS_V2.json, RANKING_V1.json, CORE_R01_ROSTER.json, R01_INSTRUMENT_V3.json: the currently published Official Core R01 benchmark data and instrument. A future Official export is covered only when it is explicitly identified as CC BY 4.0. Credit “FARTBench Core R01” and link to https://fartbench.org/data/. | CC BY 4.0 CC-BY-4.0 |
| Community review data | Anonymous Community review responses and the Community data derived from them: the current aggregate and the Community snapshots. Details below. | CC0 1.0 CC0-1.0 |
| Not covered by either | the technical report text; the website and its source code; JSON schemas and tooling; generated video media; provider-owned material; any other file on this page that is not listed above. | No licence is granted here |
Core R01 results
The machine-readable result data behind the results page, published exactly as it was frozen after the reveal. Every page figure is drawn from this one file.
-
Core R01 results (RESULTS_V2.json)
ReleasedThe revealed Core R01 result data: for every clip, each reviewer's exact answer (H01–H04, by pseudonym), the 3-of-4 consensus, N/A with its reason, the four factual outcomes, the three ratings with how many reviewers gave a number, reviewer agreement, the three AI judges' answers and calibration, and the one refusal with the provider's message. There is no composite score. Frozen as fartbench-core-r01-results-v2-revealed; defined by the schema fartbench-core-results-v2.
-
The Official Core R01 leaderboard: each configuration's Factual Score (event successes plus strict source/cause successes over four requested scenes), full-factual count, shared ranks, refusals and ratings, computed once from RESULTS_V2.json and frozen as fartbench-core-r01-overall-ranking-v1-frozen. Overall Ranking V1 was adopted after the Core R01 reveal; it was not preregistered for R01.
Community layer · not the Official results
Community review data
Rolling, anonymous answers from public blind review, and the Community results derived from them. They are released under CC0 1.0 Universal (CC0-1.0). The licence covers the Community review data only — not the site code, the technical report, the videos or any provider media. The current aggregate is served at /api/community/summary. The Official Core R01 results above are a separate, frozen release and are not changed by Community data.
Retention. Valid locked answers are kept indefinitely as benchmark evidence. FARTBench stores no reviewer name, email address, free text or IP address. If a legal, abuse or privacy requirement ever forces a removal, it will be an explicit, versioned maintenance event; published snapshots are never silently rewritten.
Snapshots. Immutable weekly Community snapshots begin once real review collection starts in production. They will be listed here, each with its date, valid reviews, anonymous sessions, ranked configurations, SHA-256 and a download. Staging and test data are never published.
Pilot-era canonical test specifications
The eight model-independent semantic specifications of the Core v0.1.0 pilot, one file per test, kept as a record. Under the pilot protocol they outranked every rendered prompt: a rendered prompt that changed the meaning of one of them was invalid. Core R01's four scenes and their questions are on the scenes page. Each row downloads the exact file whose bytes produced the digest beside it.
| File | Test | Size | SHA-256 |
|---|---|---|---|
| FB-C01.yaml | Basic Discreet Emission | 2.1 KB | 06f6319284e286476e9ffdad2d9672b3418e1deb746da683b1fb1de03bf198e4 |
| FB-C02.yaml | Multi-Agent Source Attribution | 2.4 KB | fc1d52afbcb71ea1f3afbb393480919d758a76279d5e3e9be0d49f796fa425f2 |
| FB-C03.yaml | Speech Interruption | 2.1 KB | 5ff56097fc1e6c5d9069255eaa300cf1aa4c95b0d0d18c83dc783387e04871e1 |
| FB-C04.yaml | Sustained Tonal Emission | 2.0 KB | a94251514e98e267ea22ce633b5bc0fab0ac4ecec61d7a07fe0e8386c1843f0b |
| FB-C05.yaml | Movement-Coupled Source | 2.1 KB | e5cfa3bb485136357be17846754b301d435ddd036c3b430675def0bccb4d7eb6 |
| FB-C06.yaml | Silent Condition | 2.1 KB | ece8f1b351b4ea2135ae93049b489df1b6409753705e9ab02bfd14383345294c |
| FB-C07.yaml | Exact Event Counting | 2.1 KB | 3a2bb582cf31ca4fae7bd960525e0779be8cb8d81888bdc92b98fc5c1cfead22 |
| FB-C08.yaml | Room-Acoustics Reverberation | 2.1 KB | 86a3e364b448f33907a97be6e5c8cd9496c4c8ba2ecb7b68c100d7ff30d10686 |
Other released artifacts
Digests are computed at build time from the canonical files themselves, so a published hash cannot go stale relative to the file it describes. Files labelled “Pilot” belong to the pilot era (Core v0.1.0 and R00) and are kept as a record; they are not part of the Core R01 protocol or data.
-
Core R01 human questionnaire
ReleasedThe frozen human instrument V3 as data: every scene gate, the three plain-language ratings with their anchors, the N/A reasons and the full dependency map. Byte-identical to the copy inside every reviewer package.
-
Core R01 roster
ReleasedThe first-wave configurations by name and family with their provenance basis, and the cohort cell counts. No clip links, media properties or results before the reveal.
-
One human review of one blind clip under instrument V3: gates, ratings, the N/A answer objects, playback and lock.
-
Core R01 AI evaluation record schema
ReleasedOne blind AI judge review under instrument V4, reusing the human schema’s own answer definitions.
-
Core results schema v2
ReleasedThe versioned results document the results pages are rendered from: exact answers, 3-of-4 consensus, N/A reasons, the four factual outcomes, ratings with their counts, AI calibration, refusals. No composite.
-
Core R01 instrument notes
ReleasedWhy the ratings are asked in plain words, and what stayed exactly as it was.
-
Pilot era (Core v0.1.0): the canonical specification, adapter policy, and every frozen rendered prompt for the FB-C02 commissioning wave.
-
Pilot era (Core v0.1.0): every commissioned family, configuration, provider model ID, prompt key, commissioning status, and generation setting. The Core R01 configurations are in the Core R01 roster.
-
Pilot generation manifest schema
ReleasedPilot era (R00): required provenance fields for a single generation attempt, including prompt hash, provider identity, cost, latency, moderation state, and raw output digest.
-
Pilot evaluation record schema
ReleasedPilot era (Core v0.1.0): the structure of a locked blind human evaluation, with objective observations, seven perceptual scores, failure flags, rater confidence, lock timestamp. Core R01 records use the Core R01 human evaluation record schema.
-
Pilot blind evaluation form
ReleasedPilot era (Pilot v0.1): the rater-facing instrument. Objective checks are answered before perceptual scoring, and the form is locked before model identity is revealed. Core R01 reviewers used the Core R01 human questionnaire.
-
Pilot v0.1 pack manifest
ReleasedSHA-256 digests and byte sizes for every file in the frozen pilot pack, including the three rendered FB-C02 prompt hashes.
-
Pilot era (Core v0.1.0): SHA-256 digests for the released FB-C02 v0.3 prompt-set files.
-
Technical report (PDF)
ReleasedFARTBench Technical Report FB-TR-2026-001, version 1.0. Baseline figures in the report are illustrative; see its publication notice.
Planned pilot-era result files
Files from the original Core v0.1 release plan. None has any rows; the Core R01 results are published above.
| File | Contents | State |
|---|---|---|
| generation-manifest.csv | One row per generation attempt with full provenance: provider, exact model ID, rendered-prompt hash, settings, seed, cost, latency, moderation state, raw output digest. | Not yet published |
| evaluations.csv | Locked blind human evaluation records, one row per rater per clip. | Not yet published |
| failure-events.csv | Multi-label failure-code observations joined to clips and raters. | Not yet published |
| results.json | Aggregated FARTScore, bootstrap confidence intervals, sample counts, and dimension means per configuration. | Not yet published |
| model-registry.json | Resolved model identities, including any provider-side context processors that form part of model identity. | Not yet published |
Preserved provenance
Recorded for every generation attempt, successful or not. Raw outputs are immutable evidence; a browser-compatible playback derivative may be produced but carries its own hash and stays linked to the raw artifact.
- Model family
- Exact model configuration
- Provider
- Exact provider model ID
- Prompt adapter and version
- Rendered prompt SHA-256
- Generation settings
- Generation timestamp (UTC)
- Charged or estimated cost
- Generation latency
- Seed, where exposed
- Provider request or job ID
- Raw output SHA-256
- Playback derivative SHA-256
- Moderation and refusal status
- Benchmark version
Frozen pilot prompt hashes
The three rendered prompts frozen for the nine-clip FB-C02 smoke test, with the digests recorded in the pilot pack manifest. Prompt bytes are immutable once a benchmark generation has been viewed.
| Adapter | SHA-256 |
|---|---|
| veo-3.1 | d29534ac23518301cd2135e5ad65e5efddfa5a1f457c77336f1c31fa40e91440 |
| seedance-2.5 | a1d4593cd91421dfd0365da8d80af7272fe3fa6b4ed29d821aea68ffc0bc4e12 |
| minimax-h3 | 6525cd48dfb094173420d0ee37d88930691c8c207f5e9a74d2359d554acff6de |
Licences are set out in Licences at the top of this page. A file that is not listed there carries no licence from this site.