Shway

Inspiration

I was frustrated that AI was doing too much of the creative work for me. I wanted an application that felt like an extension of me, while lowering the learning curve of complex tools like video editors.

Shway turns that idea into a way to collaborate: say “tighten this pause,” gesture through the footage, or ask for “the take where I laugh.” The creator directs the edit, and AI helps make the result concrete, inspectable, and reversible. The main hackathon work was the voice, gesture, and conversational editing experience.

What it does

Shway combines a real video editor with an AI assistant and camera-based hand controls.

• Shape the edit through conversation. Ask about the footage, trim a clip by seconds or frames, remove a section, or move a clip before or after another. The assistant uses the current timeline and transcripts to propose an actual edit.
• Direct it with your hands. Pinch and twist to shuttle through footage, use fine mode for frame steps, swipe between edit points, chop to cut, and double tap to play or pause. The interface shows the active gesture mode and stops control when tracking is lost.
• Speak a creative correction. Push-to-talk captures the current selection with the utterance. A request such as “That sounds rehearsed; use the take where I laugh” can select an alternative from indexed takes and stage a replacement.
• Start from a story brief. Describe the message, audience, and feeling; the AI chooses among transcript-indexed takes and assembles source-backed clips with linked audio into a first cut.
• Keep control of every revision. AI changes enter a preview. Accept applies the proposal as one undoable edit; reject discards it. Mouse, keyboard, and precise timeline controls remain available.

How we built it

We built Shway on the open-source FilmCraft editor rather than writing a new editing engine during the hackathon. FilmCraft supplies the Rust timeline, media pipeline, rendering, and conventional editing tools. Our contribution is the interaction surface and the validated path from creative intent to editor commands.

A TypeScript adapter connects the browser surface to the Rust engine, with a local control-channel bridge for the desktop assistant. One authoritative project owns the timeline and undo history. We added a proposal boundary that stages candidate edits separately and checks the project revision before accepting them.

MediaPipe provides hand landmarks. A deterministic gesture recognizer handles engagement, twist, swipe, tap, and chop, with dead zones, smoothing, cooldowns, and a tracking-loss latch. Gesture control runs separately from model calls, so navigation does not wait for an AI response.

FilmCraft’s optional Whisper path produces local word-timed transcripts. Gemini handles structured creative planning and chat actions; TypeSafe Jev provides a bounded take-selection path with confidence-based review. Models choose from known clips or takes. Code validates the choice and derives edit ranges from the source index rather than trusting invented timestamps or arbitrary commands. API keys stay in a local proxy.

We also explored TRIBE v2 as a separate research direction for comparing predicted cortical responses to rendered versions. The editor can consume version-bound comparison events as suggestions, subject to creator review. This is supporting research: predicted neural responses are not a measurement of engagement or video quality, and our demonstrated event integration used a replayed event rather than live TRIBE output.

Challenges we ran into

The difficult part was connecting an AI suggestion to a trustworthy edit. Source-media time and timeline time are different, linked dialogue audio must stay synchronized, and a delayed response must not overwrite a newer edit. We validated source bounds and clip identities, preserved exact engine time, and rejected stale proposals.

Gesture interaction also needs clear intent. Resting hands, incidental movement, and lost tracking should not become edits. Explicit engagement, visible modes, and bounded gesture rules make the controls easier to inspect and cancel.

Model output was another practical challenge. A flat action schema allowed missing or invented fields, so we switched to nested action-specific schemas and strict validation. Unsupported requests return an explanation; model latency and timeouts remain real prototype limitations.

Accomplishments that we're proud of

The interaction layer operates on a real timeline. In live desktop checks on a 53-clip project, we previewed, accepted, rejected, and undid trims, removals, and rearrangements while preserving linked audio. A one-second trim became a 60-frame edit at the project’s frame rate, and one undo restored the original sequence.

Browser checks demonstrated source-backed first-cut assembly and alternative-take replacement, including candidate rendering, cancel, stale-revision rejection, and one-step undo. Gesture events were also exercised against the real browser engine. These are distinct checks: live camera and microphone use across the complete demo still needs more validation.

What we learned

The interaction between the creator and AI matters as much as the model. Selection gives language a target, gestures give it immediate context, and preview plus undo makes experimentation practical. A small, validated action set can support useful creative collaboration.

What's next for Shway

Refine the live camera and microphone experience on more creator footage, measure gesture-to-visible-feedback latency, improve long-form first cuts and multi-clip rearrangement, and evaluate the research suggestions with creator and viewer feedback. The guiding goal stays the same: more creative freedom through an AI collaborator the creator can direct and correct.

Built With

• jev
• python
• rust
• tribe