Stepwise

Inspiration

Video tutorials are easy to watch, but hard to follow. You pause, rewind and look at your hands, and then back at the screen. You often find a mistake only ten steps later, when it is too late to fix. A teacher next to you finds it at once. We wanted that teacher for everyone, for any tutorial. We started with origami, because a small error in one fold makes all the next folds wrong.

What it does

Origami turns any YouTube tutorial into a personal coach: tutorials that follow you back.

• Import: Paste a YouTube link. AI watches the video and divides it into steps, one fold or action each. Each step gets a short instruction, a clip from the video, a description of the result and check points.
• Materials check: Before step 1, the app checks your materials. For example, it tells you when your paper is a rectangle and not a square, and how to fix it.
• Coach: Point a camera at your desk and start. The app speaks each step and watches your hands. After each move, it checks your work and says "Good", gives the next action or says "Stop" and tells you how to fix the mistake.
• Replay: Say "repeat" or "slow motion" to see the clip of the step in a large view. After a mistake, the app shows the correct move in slow motion automatically.
• Hands-free: You control it with your voice: "repeat", "slow motion", "next", "back".
• Any task: Origami is the first example. The same coach works for drawing, crafts, simple repairs and other hands-on tasks.

How we built it

• App: A native macOS app in SwiftUI.
• Camera: The iPhone is the camera, through Continuity Camera and Desk View, which gives a top-down view of the desk with no stand. AVFoundation supplies the frames.
• Motion detection: Core Image compares small frames 10 times each second. The app checks your work only after a real move, when your hands are still.
• Judge: Gemini 3.8 Flash gets the step, the next two steps, a photo of your work before the step, frames of your last move and the live photo. It returns a structured verdict: correct, in progress, not done, wrong or cannot see. A Nebius option (OpenAI-compatible) lets the judge also run on open-source vision models.
• Video understanding: Gemini watches the YouTube video directly and returns the steps with timestamps.
• Shape check: Apple's Vision framework measures the shape of the paper.
• Voice: ElevenLabs for natural voices, including accents. The Apple Speech framework, on the Mac, for the voice commands.
• Video playback: The YouTube IFrame API loops the clip of each step and plays it in slow motion.

Challenges we ran into

• Speed: The first checks took 5 to 16 seconds. A second check with a larger model took up to 34 seconds, so the app could not follow the person. We now use only a fast model with low thinking, and a check takes 3 to 4 seconds.
• Finding mistakes: At first, the judge said "wrong" only 2 times in 40 checks. It called a wrong fold "not done yet". We added a photo of your own work before the step, a "what changed" description and a list of the usual origami mistakes.
• Step size: The AI first combined several folds in one step, so the coach gave no feedback after your first fold. We changed the import to one fold for each step, and added an "in progress" verdict.
• Timing: The app reacted to each small hand movement, interrupted itself and missed pauses in the middle of a step. We added rules for a "real move", checks during pauses and two levels of speech.
• Fine detail: Vision models see large shapes well. A crease that is a few millimeters off position, or a mountain fold in place of a valley fold, is still hard.
• Voice loops: The microphone heard the app's own voice, and a step text can contain the word "repeat". The app now ignores commands while it speaks.
• Video choice: YouTube Shorts are too fast, at 1 to 2 seconds for each step. Good videos have a top-down camera, normal speed and spoken steps.

Accomplishments that we're proud of

• Fully hands-free: One YouTube link becomes a coach that you control only with your voice and your hands.
• Correct steps from a real video: The import found the correct steps in real tutorials, with the correct origami terms (square base, petal fold, inside reverse fold).
• Fast checks: A check takes 3 to 4 seconds, fast enough to feel like a coach that watches you.
• No special hardware: It uses an iPhone that you already have as an overhead camera.
• Not only origami: The same app also works with other tutorials, such as a pen-spinning trick.

What we learned

• The AI decides less than you think. Voice AI models only answer when you speak, so the app must decide when to look and when to speak. Simple rules in code (when the hands are still, what counts as a real move, when to repeat a hint) matter as much as the model.
• Context makes the judge accurate. A photo of your own work before the step, and the next steps, were worth more than a larger model.
• Speed is a feature. A correct answer that comes 30 seconds late is a wrong answer.
• The input decides the output. A good tutorial video gives good steps.

What's next for Origami

• Accuracy: Measure the accuracy on many folds with intentional mistakes. Compare Gemini and open-source models on the same check log.
• Natural conversation: A real-time voice, so that you can ask "what did I do wrong?" and get an answer.
• Video scores: Rate each video at import (camera angle, speed, spoken steps) and warn about bad videos.
• More devices: iPhone and iPad apps, and smart displays such as the Nest Hub.
• More tasks: A library of tested tutorials for drawing, crafts, cooking and repairs.
• Community: Share tutorials that work well with the camera check.

Built With

• apple-vision
• continuity-camera
• eleven-labs
• swiftui