FARTBench Core R01

FB-C02 v0.3 commissioning matrix

Model registry

A family is a model line; a configuration is a specific tested variant with its own provider model ID, tier, settings, and eventual results. FARTBench never pools the two.

Families
13
Configurations
25
Core AV
23
Legacy
2
Scored
0

Core AV

Native audiovisual generation in a single request.

  • Google

    Veo 3.1

    Not yet scored

    Three commissioned tiers sharing one frozen rendered prompt, allowing a quality-tier comparison within a single prompting architecture.

    Configurations
    3
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • Veo 3.1 ref
    • Veo 3.1 Fast
    • Veo 3.1 Lite

  • Google

    Gemini Omni

    Not yet scored

    Single commissioned configuration prompted with an explicit timestamped adapter rendering.

    Configurations
    1
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • Gemini Omni 1.1 Flash ref

  • ByteDance

    Seedance

    Not yet scored

    Pilot v0.1 smoke-test cell. Adapter renders the canonical timeline as explicit timestamp blocks with a single-take declaration.

    Configurations
    1
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • Seedance 2.5 ref

  • Black Forest Labs

    FLUX 3 Video

    Not yet scored

    Full and draft render modes commissioned against the same model ID and the same rendered prompt.

    Configurations
    2
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • FLUX 3 Full ref
    • FLUX 3 Draft

  • Alibaba

    Wan 3.0

    Not yet scored

    Two commissioned configurations with distinct model IDs; provider-side prompt extension is disabled where the route exposes the control.

    Configurations
    2
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • Wan 3.0 ref
    • Wan 3.0 Prime

  • Vidu

    Vidu Q3

    Not yet scored

    Pro and Turbo tiers commissioned with audio generation explicitly enabled.

    Configurations
    2
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • Vidu Q3 Pro ref
    • Vidu Q3 Turbo

  • LTX

    LTX-2.5

    Not yet scored

    Pro and Fast tiers commissioned with audio generation explicitly enabled.

    Configurations
    2
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • LTX-2.5 Pro ref
    • LTX-2.5 Fast

  • MiniMax

    MiniMax H3

    Not yet scored

    Pilot v0.1 smoke-test cell. Whether a run uses direct structured T2VA or the H3-Context-IR path is part of model identity and is recorded per sample; the two are not pooled.

    Configurations
    2
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • H3 hosted / 768P ref
    • H3 hosted / 2K

  • Kuaishou

    Kling VIDEO 3

    Not yet scored

    Up to three commissioned configurations depending on which routes the provider exposes.

    Configurations
    3
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • Kling VIDEO 3.0 Omni ref
    • Kling VIDEO 3.0
    • Kling VIDEO 3.0 Turbo
  • Not yet scored

    One API-first configuration; the Fast tier is deferred because no public API model ID has been verified.

    Configurations
    2
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • Grok Imagine Video 1.5 ref
    • Grok Imagine Video 1.5 Fast

  • HappyHorse

    HappyHorse

    Not yet scored

    Single commissioned configuration prompted with explicit timestamp blocks.

    Configurations
    1
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • HappyHorse 1.1 ref

  • PixVerse

    PixVerse

    Not yet scored

    Two commissioned configurations with separate rendered prompts; the C1 adapter explicitly suppresses multi-shot storyboarding.

    Configurations
    2
    Audio request
    Native audio
    Access
    API-first
    Weights
    Not recorded
    • PixVerse V6 ref
    • PixVerse C1

Legacy

Discontinued or historical configurations, captured for provenance and reported separately.

  • OpenAI

    Sora 2

    Legacy

    Legacy capture only, retained for provenance if access remains. Legacy configurations are reported separately from active Core AV.

    Configurations
    2
    Audio request
    Not specified
    Access
    Legacy capture if access remains
    Weights
    Not recorded
    • Sora 2 ref
    • Sora 2 Pro

Registry notes

Identity is preserved exactly
Provider model IDs and generation settings are carried verbatim from the commissioning matrix. Where the matrix records a route rather than a published identifier — for example provider-specific — the site says so rather than inventing an ID.
Context processors are part of model identity
A result produced through a provider-side preprocessing stage is not pooled with one from direct prompting unless a track explicitly defines them as the same system. MiniMax H3's Context-IR path is the clearest case and is recorded per sample.
Requested settings are not capabilities
Audio request records the generation setting the commissioning matrix specifies for a configuration. It is not a verified model capability, and no configuration has been run.
Weights status is unrecorded
The current authority does not state open- or closed-weight status for any family, so every family shows "Not recorded" rather than a guess. This field is populated when a verifiable source is added to the benchmark record.
Deferred tracks
Visual-only (SBD) and composed audiovisual pipelines are represented in the data model but are not commissioned in this release. Runway, Pika, Luma, Firefly, and Marey are not part of the current API-first matrix.

Source: benchmark/prompts/fb-c02/v0.3/FB-C02_configuration_matrix_v0.3.csv · Protocol Core v 0.1.0