One clip at a time
- The reviewer sees one clip, its blind ID, the scene's title and six questions — nothing else.
- Answers can be locked only after the clip has played through once, unmuted.
- A locked answer is written once and can never be overwritten or edited.
- The reviewer can stop at any time and resume at the next unanswered clip.
Balanced, independent orders
Each reviewer has a fixed order over the same 63 clips, generated before they start: no scene appears twice in a row, no two neighbouring clips come from the same configuration, and no two reviewers share an order. The balancing is done with hashed keys, so the order itself carries no configuration name. Suggested breaks fall after clips 21 and 42.
Reviewing on their own computers
Reviewers after the first work from a self-contained package on their own machine. It installs nothing, is bound to one pseudonymous reviewer ID, and checks its own files before it starts. Each reviewer sends back one results file, which is fingerprinted on arrival, validated against the frozen questionnaire and preserved byte for byte. Only a complete pass is used; a partial one is recorded and set aside.
Reviewers are identified only by pseudonyms. No names, email addresses or other personal details are collected with the answers.
How the reviews are combined
The primary cohort — four reviewers — and the rules for combining their answers were fixed before any real answers were combined. For every clip and every question, each reviewer's own answer stays visible. A factual question has a result only when at least 3 of the 4 reviewers give exactly the same answer: an N/A counts as the same answer only when its reason matches, and N/A is never a No. A question without such a consensus is counted as having none, never dropped. Ratings are summarised from the numbers actually given, and every rating says how many reviewers gave one; N/A answers are counted and set aside. Agreement is Krippendorff's alpha (nominal over exact answers for the factual questions; interval for ratings), always shown beside the raw votes.