Agentic Video Auto-Encoder
A self-evolving auto-encoding formulation whose two optimization stages improve Overall reconstruction by 6.6 points over removing both stages.
AVA-Encoder: Towards Agent-Native Video Representation Learning
1 Qwen Business Unit of Alibaba · 2 ShanghaiTech University
3 The Hong Kong University of Science and Technology · 4 Institute of Computing Technology · 5 Southeast University
† Project Lead
* Co-corresponding authors · jmliu1217@gmail.com · renkan@shanghaitech.edu.cn
Film space → agent space. Through self-evolving agentic video auto-encoding, AVA-Encoder autonomously learns to map tightly coupled multimodal films into agent-readable, agent-operable, and cinematically faithful Film Knowledge Graph representations.
Abstract
Agentic Video Auto-Encoding
Video creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos. A key challenge is the absence of a structured video representation that is both faithful to film content and directly usable for agentic reasoning and manipulation. To address the challenge, we propose the Agentic Video Auto-Encoder (AVA-Encoder), a novel auto-encoding framework driven by agentic self-evolution to learn agent-native video representations.
AVA-Encoder transforms a video into a Film Knowledge Graph (KG) representation and then reconstructs it back into video. This Film KG representation explicitly captures entities, events, assets, and their multimodal relationships in a structured form that can be easily understood, queried, and manipulated by agents. The reconstruction residual drives a dual-loop textual-gradient optimization framework that jointly improves the Film KG representation and the Agentic Video Encoder.
Extensive experiments show that AVA-Encoder achieves a 20.7-percentage-point absolute gain, or a 73.1% relative improvement, over the strongest external baseline. In the controlled policy-only setting, its pseudo-trained Agentic Video Encoder policy also outperforms a carefully human-tuned policy while using 74.3% fewer shot-level and 70.1% fewer keyframe-level system-prompt tokens. We release the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality Film KG representations.
Contributions
What this paper introduces
Agentic video representation benchmark
Four evaluation directions and eight fine-grained dimensions for measuring reconstruction faithfulness.
Film KG dataset and editing framework
High-quality Film KG representations and graph-based editing for creation trajectories, controllable editing, remixing, and reuse.
Method
From film space to an agent-native intermediate representation
Given an input film \(V\), the Agentic Video Encoder \(E\), governed by the complete policy \(P=(P_{\mathrm{film}},P_{\mathrm{shot}},P_{\mathrm{kf}})\), produces a Film KG \(G\). A fixed rendering pipeline that calls fixed generation models as tools then reconstructs \(\hat V\):
\[G=E(V;P), \qquad \hat V=\mathrm{Dec}(G).\]
With \(\mathrm{Dec}\) fixed, reconstruction quality measures how much film information is preserved in \(G\). Reconstruction residuals provide grounded evidence for improving the shared encoding policy and the input-specific representation at separate stages.
Multi-level Agentic Video Encoder
Film-, shot-, and keyframe-level understanding maps dense multimodal video into structured text, with each finer level receiving context from the preceding level.
Film KG Representation
Typed relations preserve hierarchy, temporal order, asset references, and cross-shot dependencies after different modalities and semantic levels have been separated.
Dual-loop Textual-Gradient Evolution
Data-Agnostic Encoding Policy Pseudo-Training improves the shared shot- and keyframe-level policies; optional Data-Dependent KG Representation Refinement later improves only the current input's KG.
Reconstruction Residual
Branch-specific failures diagnose errors and construct textual gradients, \(R_{\mathrm{reward}}\) verifies candidates, and the separate \(R_{\mathrm{eval}}\) provides common final evaluation.
Component 1
Agentic Video Encoder
Coarse-to-Fine Video Understanding
The Agentic Video Encoder first captures the global film context, then analyzes each shot with that context, and finally describes keyframe details with the corresponding shot context. This ordered injection reduces information loss when tightly coupled video content is mapped into structured text.
\[
(\mathcal C_{\mathrm{film}},\mathcal B_{\mathrm{reg}})=E_{\mathrm{film}}(V;P_{\mathrm{film}}),
\]
\[
\mathcal C_{\mathrm{shot},i}=E_{\mathrm{shot}}(s_i,\mathcal C_{\mathrm{film}},\mathcal B_{\mathrm{reg}};P_{\mathrm{shot}}),
\]
\[
\mathcal C_{\mathrm{kf},i,j}=E_{\mathrm{kf}}(f^{\mathrm{kf}}_{i,j},\mathcal C_{\mathrm{shot},i},\mathcal B_{\mathrm{reg}};P_{\mathrm{kf}}).
\]
Here \(s_i\) is shot \(i\), \(f^{\mathrm{kf}}_{i,j}\) is its \(j\)-th selected keyframe, and \(\mathcal B_{\mathrm{reg}}\) is the shared character, scene, and object registry. The complete policy is \(P=(P_{\mathrm{film}},P_{\mathrm{shot}},P_{\mathrm{kf}})\); film context and the complete registry condition both finer levels, and shot context additionally conditions each keyframe.
Component 2
Film Knowledge Graph
Text-Centered Structure with a Linked Multimodal Asset Layer
The Story–Event–Shot hierarchy and its Character, Scene, Object, Style, Camera, and Audio states store structured text only. Generated keyframes, reference images, audio, and rendered shots are kept in a linked asset layer. Typed edges preserve production, temporal, and semantic dependencies.
Component 3
Dual-Loop Textual-Gradient Evolution
Stage-Separated Optimization of Policy and Representation
The stages update different objects at different times and are not nested. First, the outer loop learns the shared shot- and keyframe-level policies before deployment. After the complete policy is frozen, the inner loop may optionally refine the current video's KG at test time. All foundation-model weights remain fixed, and \(P\) and \(G\) are never updated simultaneously.
Stage 1 · before deployment
Outer Loop
Data-Agnostic Encoding Policy Pseudo-Training
When: before deployment · Updates: shared \(P_{\mathrm{shot}}\) and \(P_{\mathrm{kf}}\) in separate branches · Freezes: \(P_{\mathrm{film}}\) and all model weights
\[
P^*=(P_{\mathrm{film}},P_{\mathrm{shot}}^*,P_{\mathrm{kf}}^*),
\qquad P_{\mathrm{film}}\ \text{fixed}.
\]
The Outer Loop applies separate propose–verify–accept branches to learn reusable shot- and keyframe-level encoding prompts from a stream of \(L\) videos—six in the reported pseudo-training setup. The resulting complete policy is shared by later, unseen video inputs.
Stage 2 · optional at test time
Inner Loop
Data-Dependent KG Representation Refinement
When: optional at test time · Updates: current video's \(G\) · Freezes: complete policy \(P^*\) and all model weights
\[
G_{V,\beta}^*=\arg\max_{G\in\mathcal G_{\mathrm{reach}}(V;P^*)}
R_{\mathrm{reward}}^{\beta}\!\left(V,\mathrm{Dec}(G)\right).
\]
The Inner Loop refines only the Film KG of the current input. Here \(\beta\in\{\mathrm{KF},\mathrm{shot}\}\) selects keyframe or video-shot refinement.
Stage 1 · before deployment
Outer Loop
Data-Agnostic Encoding Policy Pseudo-Training
Stage 2 · optional at test time
Inner Loop
Data-Dependent KG Representation Refinement
In the dual-loop schematics above, \(R(\cdot,\cdot)\) denotes reconstruction-derived loop feedback. The paper's formal optimization notation is \(R_{\mathrm{reward}}\); the separate final reporting protocol \(R_{\mathrm{eval}}\) is not used to optimize either loop.
Signal design
Reconstruction Residual
Failure evidence for textual gradients, one reward for verification, and one protocol for reporting
A complex image or video is difficult to judge with one direct score. AVA-Encoder therefore decomposes each selected keyframe or shot into approximately 30 atomic factual questions. Their answers provide precise reconstruction feedback for optimization, while a separate common protocol provides final comparison across representation systems.
Loop optimization · \(R_{\mathrm{reward}}\)
\[
R_{\mathrm{reward},n,s}(P)
=\frac{1}{K_{n,s}}\sum_{k=1}^{K_{n,s}}\chi_{n,s,k}(P)
\]
For shot \(s\) in video \(n\), \(\chi_{n,s,k}=1\) when reconstruction preserves atomic fact \(k\). Failed facts provide localized evidence for textual-gradient construction, while the scalar reward supports candidate verification and acceptance.
Final reporting · \(R_{\mathrm{eval}}\)
\[
R_{\mathrm{eval}}(V,\hat V)
=\sum_{d=1}^{D_{\mathrm{app}}}\omega_d^{\mathrm{eval}}(1-r_d^{\mathrm{eval}})
\]
Here \(r_d^{\mathrm{eval}}\) is the normalized residual for applicable film dimension \(d\), and \(\sum_d\omega_d^{\mathrm{eval}}=1\). This protocol reports V, KF, V-BC, KF-BC, and Overall; it does not optimize either loop.
Quantitative · RQ1
Does AVA-Encoder improve reconstruction fidelity?
Final evaluation uses direct Video comparison (V), direct Keyframe comparison (KF), Video Back-Captioning (V-BC), and Keyframe Back-Captioning (KF-BC). Overall is their unweighted mean.
| Method | V ↑ | KF ↑ | V-BC ↑ | KF-BC ↑ | Overall ↑ |
|---|---|---|---|---|---|
| VideoAnalyzer | 26.1 | 28.5 | 9.7 | 21.7 | 21.5 |
| Storyboard Studio | 16.4 | 28.6 | 9.6 | 23.0 | 19.4 |
| soap2soap | 36.7 | 39.5 | 15.8 | 21.3 | 28.3 |
| AVA-Encoder | 57.8 | 73.7 | 29.7 | 34.6 | 49.0 |
Main reconstruction results (%). AVA-Encoder improves Overall reconstruction by 20.7 percentage points over the strongest external baseline.
Both optimization stages versus removing both stages, equivalent to 15.6% relative improvement.
Pseudo-trained versus human-tuned policy under the controlled policy-only comparison.
Automatic rankings agree with two mutually blinded expert annotators on 97.3% of comparison triples.
RQ2 What does each optimization stage contribute?
The controlled ordering separates the two stages: (49.0) with both stages, (45.8) with policy pseudo-training only, (45.4) with KG refinement only, and (42.4) with neither. Each stage improves reconstruction on its own, and their combination performs best.
Neither stage42.4
Inner Loop only45.4
Outer Loop only45.8
Both stages49.0
Policy evolution
45.8 > 44.4The pseudo-trained policy exceeds the human-tuned policy with the Inner Loop disabled in both conditions, while reducing shot-level prompt tokens from 31,336 to 8,052 and keyframe-level prompt tokens from 13,574 to 4,062.
Acceptance gates
49.0 > 43.5The complete gated system improves Overall by 5.5 points over removing both acceptance gates.
| Ablation setting | V ↑ | KF ↑ | V-BC ↑ | KF-BC ↑ | Overall ↑ |
|---|---|---|---|---|---|
| Naive Agentic Video Encoder | 35.1 | 38.4 | 15.4 | 21.1 | 27.5 |
| Remove Data-Agnostic Encoding Policy Pseudo-Training | 52.7 | 71.8 | 23.9 | 33.1 | 45.4 |
| Remove Data-Dependent KG Representation Refinement | 55.6 | 68.3 | 26.6 | 32.7 | 45.8 |
| Remove both optimization loops | 50.7 | 67.6 | 21.2 | 29.9 | 42.4 |
| Human-tuned Agentic Video Encoder | 53.5 | 68.1 | 25.8 | 30.2 | 44.4 |
| Remove acceptance gates | 53.1 | 67.2 | 24.3 | 29.4 | 43.5 |
| AVA-Encoder (full) | 57.8 | 73.7 | 29.7 | 34.6 | 49.0 |
Reconstruction ablation results (%). The comparison directions are direct Video comparison (V), direct Keyframe comparison (KF), Video Back-Captioning (V-BC), and Keyframe Back-Captioning (KF-BC). Overall is their unweighted mean.
Qualitative reconstruction results
Reconstruction Showcases
Each comparison uses the shared generation setting. Rows show GT, AVA-Encoder, soap2soap, VideoAnalyzer, Storyboard Studio, and the naive Agentic Video Encoder.
Select any reconstruction figure to inspect it at its original resolution.
KG operability · RQ3
Graph-topology linked editing
A local graph edit follows typed dependencies to update affected character states, actions, dialogue, keyframes, and shots while preserving unrelated subgraphs. The examples show two identity-editing settings and one visual-treatment edit.
Downstream reuse · RQ4
Can the learned representation improve other video agents?
We supply the complete AVA-Encoder representation once as text, with the same basic one-sentence request to refer to it for the current case. No framework-specific adapter or prompt tuning is used. Overall story-video quality improves for all four tested systems.
MovieAgent2.47→3.30
FilmAgent1.85→2.83
Anim-Director1.85→2.50
VideoStudio1.90→1.95
Downstream story-video quality (1–4). Values compare generation without the representation to generation with a single textual injection of the complete AVA-Encoder representation.
Film KG Dataset
Text-based Film KG representations at the scale of tens of thousands of shots
We construct and release Film KG representations derived from high-quality, human-made film content at the scale of tens of thousands of shots. The dataset contains fine-grained structured descriptions of scripts, characters, scenes, objects, shots, and keyframes. It includes only the structured-text hierarchy, states, and relations—not source-film pixels or generated image, audio, or video assets. Users can connect their own generation APIs, use the representations as agentic video creation trajectories, or perform graph-topology editing.
Release
Source code
The AVA-Encoder source code is available in the official GitHub repository. The released project covers the complete framework described in the paper; consult the repository for the current availability of the evaluation toolkit, system prompts, data, and graph-editing tools.
- Project page and method overview
- Reconstruction and editing visualizations
- AVA-Encoder source code
- Reconstruction benchmark and evaluation toolkit — see repository
- Agentic Video Encoder system prompts — see repository and paper appendix
- Film KG Dataset and graph-editing tools — see repository
Paper and citation
AVA-Encoder: Towards Agent-Native Video Representation Learning
Paper page: https://arxiv.org/abs/2608.12313
BibTeX
@misc{li2026avaencoder,
title = {AVA-Encoder: Towards Agent-Native Video Representation Learning},
author = {Chuyue Li and Jinpeng Yu and Haozhe Wang and Tian Xueyun and
Zhijing Zhang and Bingnan Li and Shuqi Gu and Kan Ren and
Jiaming Liu and Ruihua Huang},
year = {2026}
}