This AI Video Model Plans Every Shot in Code First, Like Previs
A Microsoft Research team’s Code Video Model builds every AI shot as Three.js previs first, then renders it with MiniMax-H3 for control over camera, timing, and physics.
A Microsoft Research team has released Code Video Model, a research system that plans AI video shots in code before generating them. A coding agent builds each shot as a simple 3D scene, and a video model turns that rough previs into the final clip.
The approach puts layout, camera paths, timing, and physics into editable code instead of prompts. Changing a shot can be as small as editing one line.
How Code Video Model Works
The system starts with a text prompt. GPT-6 writes a Three.js program that lays out the scene, moves the objects, places the camera, and calculates the physics from equations.
That program renders a low-poly proxy video in under a minute. A separate reference image sets the look of the final shot.
MiniMax-H3 then generates the final clip from both. The proxy provides structure and motion, and the reference image provides materials, lighting, and detail.
Borrowed From Film Previs
The team describes the idea as borrowed from film previs. Project lead Ziyu Wan and researcher Gordon Guocheng Qian introduced the work on X, and Qian summed up the approach as code to previs, inspect, revise, then video.
Because the whole shot lives in one program, it can be revised the way a previs artist would revise a scene. In one example, a robot’s path changes by editing its route waypoints in a single line of code, while the rest of the shot stays the same.
The proxy can be checked and adjusted before any expensive video generation happens. That’s the same reason productions use previs in the first place.
Why Code Helps With Physics and Control
The researchers point to two gaps in current video models: physics and precise control. A thrown ball might hover, a pendulum might drift off its timing, or a collision might look plausible while breaking physical laws.
Calculating the motion in code first gives the video model a physically grounded plan to follow. The project’s examples include bullet time, product cinematography, architectural walkthroughs, robotics simulation, and 3D and 4D reconstruction.
The Early-Step Trick
Feeding a crude low-poly proxy straight into a video model tends to produce crude, low-poly results. The team’s fix is to use the proxy only during roughly the first 25% of the generation process.
That early guidance locks in layout, viewpoint, and motion. After that the proxy is removed, and the reference image and the model’s own knowledge handle appearance.
The method is training-free. MiniMax-H3 isn’t modified, and no extra control module is trained.
What’s Available
The code is on GitHub under an Apache 2.0 license, and a 22-case benchmark called CodeVideoBench is on Hugging Face. Running it requires Ubuntu, an NVIDIA GPU with CUDA, the MiniMax-H3 checkpoint, and an OpenRouter API key for the GPT-powered steps.
The demo clips are five seconds at 960x544. There’s no app or hosted demo yet, so for now this is a research release aimed at developers and researchers.
Competitive Context
Using rough 3D to steer AI video isn’t new. H3 Max already supports 3D-to-video workflows from Blender previs, covered in H3 Max Turns Blender Previs Into Video.
Code Video Model goes a step further by having an AI agent write the 3D scene from a text prompt. That puts it close to the code-driven video tools covered in What Is HTML-to-Video?, where agents write motion graphics as code.
On the team’s own benchmark, Code Video Model scored 85.2, 82.1, and 85.8 across three evaluator models. That compares with 74.3, 73.1, and 80.6 for VACE, the strongest learned baseline they tested.
The Signal in the Noise
This won’t change anyone’s workflow this week, since it needs a Linux machine with an NVIDIA GPU and some coding comfort. The idea is the part to pay attention to.
Previs gives directors exact control over blocking, camera moves, and timing before they commit to a shot. Code Video Model brings that same structure to AI video, and the edit-one-line workflow fits naturally with AI agents.
The paper is upfront about the limits. Frame-exact alignment is still hard, and the reference image doesn’t always win when it conflicts with the proxy or the model’s defaults.
The Details
- Developer: Microsoft Research team (Zhiyang Liang, Jiawei Zhang, Gordon Guocheng Qian, Dongdong Chen, Ziyu Wan)
- Scene code: Three.js, written by GPT-6
- Video model: MiniMax-H3 (frozen, no fine-tuning)
- Output tested: 5 seconds, 960x544
- License: Apache 2.0 (code)
- Requirements: Ubuntu 22.04, NVIDIA GPU with CUDA 12.6, OpenRouter API key
- Price: Free and open source; API and compute costs apply
Would you plan AI shots in a previs-style 3D layout first if the tools made it easy? Share how you’re getting camera control out of AI video right now in the comments.
