Genjutsu AI — Devpost project page

Inspiration

Replacing a person in a video is normally a reshoot. There is no version of "put a different actor in this scene" that does not involve getting everyone back on set — and once the video exists, the performance is baked in.

The interesting question is what happens if you treat motion and appearance as separable. The movement in a video is data: it is the pose and expression of a person over time. The identity is a different thing. If you can extract the first and apply it to the second, you can change who is in the video without touching how they move.

Genjutsu AI is built on that separation.

What it does

Rewrite footage without reshooting it.

• Transfer motion from a filmed performance to a new character
• Swap a person or object in existing video
• Restyle an entire scene while keeping the timing
• Preserve the original camera work

Everything is built so that the original movement and camera stay exactly as filmed — because a swap that changes the motion reads as fake immediately.

How I built it

1. Motion extraction. Per frame, extract the driving signal: body pose, face landmarks, and expression parameters. This is the representation the rest of the pipeline works on. Getting this stable is the foundation — jittery motion extraction produces jittery output no matter what happens downstream.

2. Identity conditioning. The target character is provided as an image or a few images, and encoded into an identity representation injected into the generation. This is what keeps the new character consistent across frames; without a persistent identity signal the face drifts within a few seconds.

3. Temporally aware generation. Frames cannot be generated independently — that produces flicker and identity drift. Generation happens over windows of frames with the temporal context available, and the windows are overlapped so the seams do not show.

4. Compositing back into the original. The generated region is composited into the source footage, with the background, camera motion, and lighting of the original preserved. This is what makes the result look filmed rather than pasted.

5. Long video handling. A clip longer than a few seconds is processed as a sequence of overlapping windows with consistent identity conditioning across all of them, so a two-minute clip does not become twenty different people.

Challenges I ran into

Identity drifts over time. Each frame is individually convincing and the character slowly becomes someone else. This is the central technical problem of the whole product, and it comes from conditioning being applied per-frame rather than maintained across the sequence. Anchoring identity across windows, not just within them, was the fix.

Temporal flicker is more noticeable than detail loss. A slightly soft frame that is stable over time reads as real footage. A sharp frame that shimmers reads as generated instantly. I deliberately traded resolution for temporal consistency, which felt wrong at the time and was correct.

Compositing is where it succeeds or fails. A perfect generation pasted into a shot with different lighting looks worse than an imperfect generation matched to the scene. Matching grain, colour, and motion blur before compositing mattered more than improving the model.

Accomplishments that I'm proud of

• Keeping motion and identity genuinely separate, which is what makes the transformations believable.
• Temporal stability over several seconds — the hardest part of video generation and the part users notice first.
• Preserving the original camera work, so the output looks like the same shot rather than a new one.

What I learned

In video, consistency beats quality. Every decision I made toward a more stable output improved how real it looked, even when it made individual frames worse. The eye forgives softness and does not forgive flicker.

Compositing is half the product. I spent most of my early effort on generation and was consistently disappointed by results that were technically good and looked wrong. The gap was matching the generated element to the plate.

What's next for Genjutsu AI

• Multi-person scenes. Currently one subject at a time; handling interactions between two people is the obvious next problem.
• Mask-based control. Let users specify exactly what is replaced rather than the system deciding.
• Longer clips with maintained identity. Push the stable window from seconds to minutes.
• Editable motion. Adjust the performance — timing, intensity — rather than only transferring it as filmed.

Built With

• react