--- schema: 1 type: video title: >- Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta description: >- A video is about sports for its first six seconds, then turns political for half a second. Catching that across a hundred million plus videos is one of two problems Aditya Gautam works on at Meta; the other is unoriginal content, trivial to make with AI tools and corrosive to attribution. Both sit on messy data: adversarial uploads, multilingual on screen text, drift, and no ground truth. Mismatch across modalities is a solved embedding problem; mismatch within one modality over time is not, so three agents share it. A perceiver splits the video where temporal change happens, not at a fixed frame rate, and emits clip level embeddings, tags, and OCR. A reviewer runs the temporal analysis over that JSON and folds in live comments and sentiment. A retriever indexes topics, embeddings, and entities into inverted, vector, and graph stores for pulling similar clips and authors at inference time. Every agent runs on a small specialized VLM, not a frontier model, since he does not care whether it can code. That means pretraining the vision encoder on in house images because user generated content looks nothing like clean web data, instruction tuning to a JSON schema, and a DPO loop where daily production samples pass through an in house LLM judge and a human queue tracing failures to the tool call or retrieval step. Distillation and quantization then produce a table of model sizes. Evaluation goes beyond precision and recall to per node latency, reasoning budgets, token cost, and judge drift. Three optimizations keep most videos out of the pipeline: compressing similar frames, caching verdicts on viral content, and pruning by metadata so trusted creators skip it. Speaker info: - https://www.linkedin.com/in/aditya-gautam-68233a30/ Timestamps: 0:00 - Two problems on short form video at 100 million plus scale 1:20 - Messy data: adversarial, multilingual, drifting, no ground truth 2:03 - Modality misalignment inside a single video 3:01 - Unoriginal content, and why one agent is not enough 4:22 - Reviewer, perceiver, retriever 5:31 - Temporal analysis, then indexing into inverted, vector, and graph stores 9:37 - The reviewer adds live user signals 10:45 - Pretraining and instruction tuning specialized VLMs 13:28 - DPO from production samples with humans in the queue 16:05 - Distillation and quantization to make it scalable 17:13 - Holistic evaluation: nodes, reasoning budgets, drift 19:28 - Spatial temporal reduction, caching, metadata pruning 20:35 - Takeaways --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/jNE8No-wvok-youtube-ffb117fff1dbfb39cc7d Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/jNE8No-wvok' markdown: 'https://liquidgold.cc/watch/jNE8No-wvok.md' llms: 'https://liquidgold.cc/watch/jNE8No-wvok/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:20:56.349Z' language: null children: [] videoId: jNE8No-wvok durationSeconds: 1292 publishedAt: '2026-09-23T14:00:34.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 123 endSeconds: 135 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=123' heldReason: rights - ordinal: 1 text: null startSeconds: 135 endSeconds: 190 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=135' heldReason: rights - ordinal: 2 text: null startSeconds: 335 endSeconds: 347 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=335' heldReason: rights - ordinal: 3 text: null startSeconds: 347 endSeconds: 581 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=347' heldReason: rights - ordinal: 4 text: null startSeconds: 591 endSeconds: 645 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=591' heldReason: rights - ordinal: 5 text: null startSeconds: 657 endSeconds: 813 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=657' heldReason: rights - ordinal: 6 text: null startSeconds: 824 endSeconds: 975 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=824' heldReason: rights - ordinal: 7 text: null startSeconds: 985 endSeconds: 1040 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=985' heldReason: rights - ordinal: 8 text: null startSeconds: 1050 endSeconds: 1170 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=1050' heldReason: rights - ordinal: 9 text: null startSeconds: 1170 endSeconds: 1181 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=1170' heldReason: rights - ordinal: 10 text: null startSeconds: 1181 endSeconds: 1239 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=1181' heldReason: rights - ordinal: 11 text: null startSeconds: 1181 endSeconds: 1239 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=1181' heldReason: rights - ordinal: 12 text: null startSeconds: 190 endSeconds: 202 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=190' heldReason: rights - ordinal: 13 text: null startSeconds: 202 endSeconds: 266 watch: 'https://liquidgold.cc/watch/jNE8No-wvok?t=202' heldReason: rights --- # Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta A video is about sports for its first six seconds, then turns political for half a second. Catching that across a hundred million plus videos is one of two problems Aditya Gautam works on at Meta; the other is unoriginal content, trivial to make with AI tools and corrosive to attribution. Both sit on messy data: adversarial uploads, multilingual on screen text, drift, and no ground truth. Mismatch across modalities is a solved embedding problem; mismatch within one modality over time is not, so three agents share it. A perceiver splits the video where temporal change happens, not at a fixed frame rate, and emits clip level embeddings, tags, and OCR. A reviewer runs the temporal analysis over that JSON and folds in live comments and sentiment. A retriever indexes topics, embeddings, and entities into inverted, vector, and graph stores for pulling similar clips and authors at inference time. Every agent runs on a small specialized VLM, not a frontier model, since he does not care whether it can code. That means pretraining the vision encoder on in house images because user generated content looks nothing like clean web data, instruction tuning to a JSON schema, and a DPO loop where daily production samples pass through an in house LLM judge and a human queue tracing failures to the tool call or retrieval step. Distillation and quantization then produce a table of model sizes. Evaluation goes beyond precision and recall to per node latency, reasoning budgets, token cost, and judge drift. Three optimizations keep most videos out of the pipeline: compressing similar frames, caching verdicts on viral content, and pruning by metadata so trusted creators skip it. Speaker info: - https://www.linkedin.com/in/aditya-gautam-68233a30/ Timestamps: 0:00 - Two problems on short form video at 100 million plus scale 1:20 - Messy data: adversarial, multilingual, drifting, no ground truth 2:03 - Modality misalignment inside a single video 3:01 - Unoriginal content, and why one agent is not enough 4:22 - Reviewer, perceiver, retriever 5:31 - Temporal analysis, then indexing into inverted, vector, and graph stores 9:37 - The reviewer adds live user signals 10:45 - Pretraining and instruction tuning specialized VLMs 13:28 - DPO from production samples with humans in the queue 16:05 - Distillation and quantization to make it scalable 17:13 - Holistic evaluation: nodes, reasoning budgets, drift 19:28 - Spatial temporal reduction, caching, metadata pruning 20:35 - Takeaways --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/jNE8No-wvok-youtube-ffb117fff1dbfb39cc7d Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/jNE8No-wvok · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.