DemoTrace

Inspiration

Hackathon judges, investors and customers often meet a product through a short video, not the product itself. The rules of this hackathon say it plainly: judges may score a project from its video and description without running the code. Watching demos (including my own), I kept seeing the same gap: the narrator says a feature works, the cursor drifts toward a button, and the video moves on. Nobody ever sees the result. I wanted a tool that tells you, before you submit, which of your claims the video actually proves.

What it does

You upload a demo video (MP4, up to 3 minutes) and write 3–5 observable claims, such as "Clear completed removes completed tasks." DemoTrace returns an evidence map:

• Shown: the result is visible on screen.
• Told only: the claim is said, or promised on a slide, but never demonstrated.
• Missing: neither said nor shown.

Every verdict links to timestamped evidence. Click it and your local video jumps to that exact moment, so you can verify it yourself. When a claim is only told, you know exactly what to re-record. Re-check keeps the same claims, so you can test the fixed recording against the same criteria.

How I built it

I built it with the Devpost Learn Skill Pack, letting the agent interview me before any code was written: scope.md → prd.md → spec.md, then a 12-slice build checklist where each slice was verified and committed before the next one started.

• Frontend: Next.js on Vercel. The video never leaves the browser for playback; evidence clicks seek the local file.
• Control API: an AWS Lambda Function URL with per-job bearer tokens, presigned S3 POST uploads (size and type enforced), and a DynamoDB state machine with conditional transitions.
• Worker: an arm64 Lambda container running ffmpeg and faster-whisper. It extracts sampled frames and a timed transcript, then asks Claude (Anthropic API) for a structured verdict per claim, citing frame and speech IDs.
• Trust boundary: Claude only suggests. DemoTrace validates every cited ID and computes every timestamp itself before a verdict is shown. The video is deleted from S3 right after download.
• Infra: Terraform with remote state and a GitHub Actions OIDC deploy pinned to an image digest.

Challenges I ran into

• Model access: Bedrock was blocked on my AWS account, so I moved to the direct Anthropic API, with the key kept in Secrets Manager.
• Invisible failures: every error first surfaced as "model unavailable." Attributing failures to their pipeline stage revealed the real causes.
• Deploys that never deployed: CI tested with a different ffmpeg than production, built an arm64 image on an amd64 runner, and trusted the wrong OIDC subject. I fixed these one at a time until the first green deploy.
• The hard part, shown vs told: promise slides, hovered-but-not-clicked buttons and half-shown compound claims ("create and share") all tempt a false Shown. A 0.7-second save toast fell between 1-second samples, so short videos are now sampled every 0.5 seconds.
• Bugs found by using it: while recording this video I discovered that a trailing slash in the API URL broke every upload, and that reloading mid-analysis showed the sample video instead of yours. Both are fixed.

Accomplishments that I'm proud of

A hand-labelled smoke eval of six real recordings (27 claims) run through the live Lambda → Claude pipeline, with deliberately tricky negative cases: 100% status accuracy, 100% evidence hits, and zero false Shown (0 of 16 claims that should not be Shown). It is a small eval, not a general accuracy claim, but every number comes from genuine runs.

What I learned

• Planning before building works: the scope interview cut the idea down to one complete loop instead of a pile of half features.
• Evals need negative cases. A model that says "Shown" too easily looks great on happy-path demos.
• Verify on the real system. My worst bugs only appeared in production, in the browser, under real use.
• An AI verdict is only as trustworthy as the evidence you can click.

What's next

Longer videos, multi-language narration, a shareable report link, and a CLI so teams can check every product walkthrough before it ships.

Built With

• amazon-web-services
• claude
• faster-whisper
• ffmpeg
• github-actions
• python
• terraform
• typescript