---
schema: 1
type: video
title: 'Evaling Video Slop — Maor Bril, Character.ai'
description: >-
  A generated clip where the character stands frozen for four seconds can still
  score well, because the judge rewarded the gloss and the vibe instead of what
  actually happened. That failure is the whole problem with evaling video: CLIP
  score misses temporal incoherence, a team watching clips on Friday does not
  scale, and any AI judge you wire up drifts from human preference unless you
  measure the drift. Video breaks the text playbook because it has to hold
  temporal consistency, shot continuity, and a coherent story across frames, not
  just look good in a single still. The fix that stuck was to stop scoring and
  start comparing. Absolute scores collapsed to one dimension, but pairwise
  preference, is B a better story than A, held up, so Maor Bril's team trained a
  Qwen3-VL judge with Bradley-Terry loss on pairs of real and deliberately
  broken footage to catch slop before it ships. Drift is cheapest to catch
  early, especially on longer form video, so the judge runs as a regression gate
  in CI: every AgentX release at Character.ai clears an eval wall, calibrated
  against human scores, before users ever see it. Speaker info: -
  https://x.com/maorbril - https://www.linkedin.com/in/maorbril -
  https://github.com/character-ai/judgejudy Timestamps: 0:00 - Introduction:
  evaluating AI generated video 1:19 - Why video generation drifts between
  frames 3:14 - Story and sound: what a clip has to get right 4:43 - LLM as a
  judge, and catching drift early 7:01 - Story and sound failure modes 8:28 -
  Small model vs bigger model as judge 9:20 - Don't score, compare: pairwise
  preference 10:47 - When the judge scores vibe over substance 11:53 - Pairing
  real footage to train a quality detector 13:27 - Self verification in the
  generation loop 15:05 - Q&A
canonical: 'https://liquidgold.cc/watch/b_PmGocP4rc'
markdown: 'https://liquidgold.cc/watch/b_PmGocP4rc.md'
llms: 'https://liquidgold.cc/watch/b_PmGocP4rc/llms.txt'
parent: 'https://liquidgold.cc/channel/aidotengineer.md'
channel: aidotengineer
updated: '2026-10-24T06:35:50.129Z'
language: null
children: []
videoId: b_PmGocP4rc
durationSeconds: 1393
publishedAt: '2026-07-25T00:00:02.000Z'
searchChunks:
  - ordinal: 0
    text: null
    startSeconds: 24
    endSeconds: 87
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=24'
    heldReason: rights
  - ordinal: 1
    text: null
    startSeconds: 97
    endSeconds: 198
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=97'
    heldReason: rights
  - ordinal: 2
    text: null
    startSeconds: 97
    endSeconds: 198
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=97'
    heldReason: rights
  - ordinal: 3
    text: null
    startSeconds: 97
    endSeconds: 198
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=97'
    heldReason: rights
  - ordinal: 4
    text: null
    startSeconds: 198
    endSeconds: 209
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=198'
    heldReason: rights
  - ordinal: 5
    text: null
    startSeconds: 198
    endSeconds: 209
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=198'
    heldReason: rights
  - ordinal: 6
    text: null
    startSeconds: 209
    endSeconds: 289
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=209'
    heldReason: rights
  - ordinal: 7
    text: null
    startSeconds: 209
    endSeconds: 289
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=209'
    heldReason: rights
  - ordinal: 8
    text: null
    startSeconds: 289
    endSeconds: 299
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=289'
    heldReason: rights
  - ordinal: 9
    text: null
    startSeconds: 299
    endSeconds: 433
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=299'
    heldReason: rights
  - ordinal: 10
    text: null
    startSeconds: 299
    endSeconds: 433
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=299'
    heldReason: rights
  - ordinal: 11
    text: null
    startSeconds: 444
    endSeconds: 511
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=444'
    heldReason: rights
  - ordinal: 12
    text: null
    startSeconds: 511
    endSeconds: 524
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=511'
    heldReason: rights
  - ordinal: 13
    text: null
    startSeconds: 524
    endSeconds: 567
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=524'
    heldReason: rights
  - ordinal: 14
    text: null
    startSeconds: 579
    endSeconds: 648
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=579'
    heldReason: rights
  - ordinal: 15
    text: null
    startSeconds: 579
    endSeconds: 648
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=579'
    heldReason: rights
  - ordinal: 16
    text: null
    startSeconds: 659
    endSeconds: 723
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=659'
    heldReason: rights
  - ordinal: 17
    text: null
    startSeconds: 723
    endSeconds: 735
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=723'
    heldReason: rights
  - ordinal: 18
    text: null
    startSeconds: 735
    endSeconds: 815
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=735'
    heldReason: rights
  - ordinal: 19
    text: null
    startSeconds: 918
    endSeconds: 1002
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=918'
    heldReason: rights
  - ordinal: 20
    text: null
    startSeconds: 1012
    endSeconds: 1109
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=1012'
    heldReason: rights
  - ordinal: 21
    text: null
    startSeconds: 1109
    endSeconds: 1120
    watch: 'https://liquidgold.cc/watch/b_PmGocP4rc?t=1109'
    heldReason: rights
---


# Evaling Video Slop — Maor Bril, Character.ai

A generated clip where the character stands frozen for four seconds can still score well, because the judge rewarded the gloss and the vibe instead of what actually happened. That failure is the whole problem with evaling video: CLIP score misses temporal incoherence, a team watching clips on Friday does not scale, and any AI judge you wire up drifts from human preference unless you measure the drift. Video breaks the text playbook because it has to hold temporal consistency, shot continuity, and a coherent story across frames, not just look good in a single still. The fix that stuck was to stop scoring and start comparing. Absolute scores collapsed to one dimension, but pairwise preference, is B a better story than A, held up, so Maor Bril's team trained a Qwen3-VL judge with Bradley-Terry loss on pairs of real and deliberately broken footage to catch slop before it ships. Drift is cheapest to catch early, especially on longer form video, so the judge runs as a regression gate in CI: every AgentX release at Character.ai clears an eval wall, calibrated against human scores, before users ever see it. Speaker info: - https://x.com/maorbril - https://www.linkedin.com/in/maorbril - https://github.com/character-ai/judgejudy Timestamps: 0:00 - Introduction: evaluating AI generated video 1:19 - Why video generation drifts between frames 3:14 - Story and sound: what a clip has to get right 4:43 - LLM as a judge, and catching drift early 7:01 - Story and sound failure modes 8:28 - Small model vs bigger model as judge 9:20 - Don't score, compare: pairwise preference 10:47 - When the judge scores vibe over substance 11:53 - Pairing real footage to train a quality detector 13:27 - Self verification in the generation loop 15:05 - Q\&A

Canonical: https://liquidgold.cc/watch/b_PmGocP4rc · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md)

> Search chunk text is held until public-source provenance is verified.

## Document chunks

> Missing: no stored document chunks are available.