Vidu S2 Lets You Talk to, Direct, and Edit AI Video in Real Time
Vidu S2 lets you talk to a live AI character or edit an incoming video stream in real time, both responding to natural conversation.
Shengshu Technology and Tsinghua University released Vidu S2, a real-time video model built around two distinct capabilities: an interactive character you can talk to and direct with your voice, and a live editor that changes an incoming video stream's style, outfit, or subject on the fly. The most interesting part isn't either capability alone, it's that both respond to natural conversation rather than a prompt-and-wait cycle.
What "Talking to Your Video" Actually Means
Vidu S2-Avatar generates a real-time interactive character at 720p, 25 to 42 frames per second, that responds to spoken instructions the way a person would. Say "dance for me" and the character performs full-body movement, including complex motion like dancing, something the team specifically added new training data to support in this version.
The genuinely useful part is mid-conversation flexibility: you can introduce a new reference image at any point during an active session, "put on this hat," and the character updates accordingly without restarting the interaction. That's a meaningfully different interaction model than typing a prompt and waiting for a finished clip, it's closer to giving live direction to a performer than submitting a generation request.
Editing a Video While It's Still Playing
Vidu S2-Editing applies that same real-time, conversational logic to existing footage instead of a generated character. Feed it an incoming video stream, and it can change visual style, swap an outfit, replace a character, or replace the background, all while preserving the original motion, in real time as the stream plays.
The team's own claim here is genuinely notable: in their benchmark testing, this real-time editor outperformed the offline editing models they evaluated it against. That's a real reversal of the usual assumption that real-time processing trades away quality for speed. Whether that holds up outside the team's own benchmarks is worth watching for, but it's a specific, checkable claim rather than vague marketing language.
How It Avoids Drifting Over Long Sessions
A real technical problem with any streaming or continuous generation system is error accumulation, small mistakes compounding the longer a session runs. Vidu S2 addresses this with something the team calls Self-Replay Forcing (SRF): the model replays its own previously generated output, adds fresh noise to it, and processes that through a gradient-enabled pass during training. In plain terms, it's specifically trained on its own past mistakes being reintroduced, which helps prevent those errors from snowballing across a long, continuous stream.
Where This Sits in the Broader Race
Vidu S2 builds directly on Vidu S1, released in July, upgrading resolution from 540p to 720p and adding the dynamic reference-swapping and stronger instruction-following described above. It also explores real-time spatial (stereoscopic) video generation for both the avatar and editing models, aimed at VR applications specifically.
This joins Runway's GWM Worlds 2, MiniMax's H3-World, and Alibaba's Happy Oyster as another major, well-resourced entry into real-time interactive video generation this month. Vidu's specific angle within that crowded field is conversational control, voice-directed characters and live-edited streams, rather than the keyboard/camera-control framing most of the other entries lead with.
Competitive Context
Each of these interactive video projects is solving a genuinely different piece of the same broader problem. Runway's approach splits into scripted, turn-based, and real-time authoring modes. MiniMax's H3-World retrained a tiny fraction of an existing model to add keyboard control.
Alibaba's Happy Oyster emphasizes open-ended world exploration with joint audio-video generation. Vidu S2's distinct contribution is treating both character performance and video editing as things you direct through natural conversation, in real time, rather than through a keyboard, a script, or a single prompt.
The Signal in the Noise
The interesting part here isn't the resolution bump or the benchmark numbers, it's that both halves of Vidu S2 are built around the same underlying idea: talking to your footage and having it respond immediately, whether that footage is a fully generated character or something you fed in yourself. That's a genuinely different mental model for working with AI video than typing a description and reviewing what comes back later.
Would being able to talk directly to a live-editing model, changing an outfit or a background mid-stream through conversation, change how you'd think about revising a rough cut compared to describing changes in a text prompt and waiting?
The Details
- Release: Vidu S2, by Shengshu Technology and Tsinghua University
- S2-Avatar: real-time interactive character generation, 720p at 25-42 FPS, voice-directed control, mid-session reference image updates, expanded instruction following (including full-body dance motion)
- S2-Editing: real-time video stream editing, style transfer, virtual try-on, character replacement, background replacement, preserves original motion
- Stability technique: Self-Replay Forcing (SRF), trains on re-noised self-generated output to prevent error accumulation across long streams
- Also explored: real-time spatial (stereoscopic) video generation for VR, for both models
- Builds on: Vidu S1 (July 2026, 540p); upgraded to 720p in S2
- Access: live playable demo at vidu.com/vidu-stream, plus a developer API
