göz | Devpost

Inspiration

We wanted to explore whether a video could respond to what a viewer actually pays attention to. Instead of asking someone to choose the next scene, GÖZ uses gaze as an input, with experimental EEG as a second signal. The goal is to make short-form video more responsive while being honest about what today’s prototype can and cannot infer.

What it does

GÖZ takes a short video clip and uses viewer gaze to guide the next generated segment. The current prototype can generate a six-second clip without buffering in the demo flow. An early evidence-freeze step at about 3.5 seconds can trigger the next clip without waiting for the full analysis. Eye tracking is working; raw Muse 2 signals are visible, but their quality is clipped and EEG handling still has a timeout issue to fix. EEG is experimental and is not yet a dependable control signal.

How we built it

We built a Python and FastAPI backend with a plain JavaScript frontend. The pipeline uses gaze data from Gazekit with an external phone camera, Muse 2 data through MuseLSL and Bleak, Florence vision, FFmpeg, and Fal’s MiniMax H3 Max Turbo for clip generation. We are testing a local OpenCV reference-matching path using SIFT/RANSAC and optical flow. The EEG-to-decision mapping is independently inspired by ideas we explored in Fable; this is not a claim of copying or using Fable’s code.

Challenges and what we learned

The clearest lesson was to test the real interaction loop instead of trusting the model’s prompt alone. Across seven prompt submissions, the generated results remained unchanged because character detection was not adequate. Eye tracking works, but the Muse 2 signal is clipped and an EEG timeout fix still needs a reload. We are working on local vision and an EEG decision mapping, so those parts are not validated yet. We also learned to freeze an early gaze-evidence window so the next clip can start without waiting for all analysis.

What is next

We are working to improve character detection and the local vision path, fix the EEG timeout and signal quality issues, then test whether gaze and EEG can reliably influence the generated continuation. We still need to verify the complete experience with a recorded end-to-end demo.