Links
The GitHub repo: https://github.com/UpsideDownTurtle/walkthru
The live app: https://walkthru-859455343181.us-central1.run.app
Inspiration
Picture a student moving out of their first apartment. The landlord keeps \$400 of the deposit for a carpet stain and a cracked blind that were already there on move-in day. The student has a few blurry photos with no dates on them, so it's their word against the landlord's, and the landlord has the money.
Most renters never document a place properly when they move in. Nobody hands you a checklist, and you're busy carrying boxes. I wanted something that takes one minute and works for the person who actually pays the deposit.
What it does
You walk through your new apartment with your phone recording and talk about what you see. Walkthru sends the video and your voice to Gemini, which lists each defect with a timestamp and a quote of what you said. Then it checks every one against a real frame from the video and draws a box around it. If it can't find it, it doesn't count.
It also writes the letter to your landlord, adds reminders, and finds tenant help nearby. At move-out you film again, and it tells you what was already there and what's new. You can share a read-only link with the evidence. Nothing gets sent unless you send it.
How we built it
I built it solo. It's Next.js and TypeScript on Cloud Run, with ffmpeg in the container, and Gemini through the newer Interactions API does most of the work.
The first call gets the whole video, picture and sound together, and streams back JSON. I parse that JSON while it's still arriving, so issue cards show up before Gemini has finished watching.
The second pass is the frame check. For an issue at time t, ffmpeg pulls three frames at
$$t - 1\,\text{s}, \quad t, \quad t + 1\,\text{s}$$
and Gemini picks the clearest one and returns a bounding box, or says it can't see the problem. The box comes back on a 0 to 1000 scale, so for a frame W pixels wide and H pixels tall:
$$x_{px} = \frac{x}{1000}\,W, \qquad y_{px} = \frac{y}{1000}\,H$$
Anything that fails goes to a review list and never makes it into a letter.
After that, an agent with five tools (research_deposit_law, compute_deadline, find_local_help, save_document, add_reminder) builds the case. It decides for itself which tools to call and in what order, and the app shows you every call. For move-out, the old evidence frames and the new video go into one request so Gemini can compare them directly. It can also read a photo of a landlord's deduction letter and check each charge against the evidence.
Places API (New) handles the address field and finds tenant unions and legal aid, and the Maps JavaScript API puts them on a map. Firebase does sign-in, Firestore holds the reports, Cloud Storage holds the videos and frames, and security rules keep every report private unless you share it. Reminders are Google Calendar links. There are 133 unit tests, and nothing got committed without typecheck, lint and the build passing.
Challenges we ran into
Timestamps drift when the camera is moving. The moment Gemini names is often a second off from where the defect is actually visible. Grabbing frames on both sides and letting the model pick fixed that, and it turned into the verification step.
Rendering JSON that isn't finished yet was annoying. I re-parse the partial document on every chunk and only show an issue once the next one has started, so you never see half a sentence.
My first real video call failed with "The value 'text' is not supported for 'input[1].type' in v1, it is available in v1beta". The SDK defaulted to v1, and mixed video and text input only works on v1beta. Pinning the version fixed it.
I planned to look up deposit law with Google Search grounding. That needs a paid tier, and my event credit couldn't pay for one, so every grounded call came back with a quota error. The agent still calls the same law tool, but now it reads from a small reference table bundled with the app, and the law card says so. It also says it isn't legal advice.
The free tier caps some models at 20 requests a day, and one full report uses a lot of those. I cut parallel frame checks from four to two and switched to a model with a bigger daily quota.
Accomplishments that we're proud of
On a real 28-second phone video, the first issue showed up 21 seconds after analysis started, and the whole report was done at 22 seconds. The agent puts together the law summary, letters, reminders and local help by choosing its own tool calls. And it all runs in a phone browser with no install and no sign-up.
What we learned
A multimodal model gets a lot more trustworthy when a second, narrower call has to check the first one. Streaming structured output also turned out to matter for the product, since watching issues appear one at a time feels much faster than waiting for a finished list.
I also learned how many Google pieces one small app needs: the Interactions API, Places and Maps with separate restricted keys, Firebase Auth, Firestore and Storage with rules doing the access control, and Cloud Run deploys from source. Cost and quota shaped the design from the first day. And when a feature can't work, a fallback that says what it's doing is better than one that breaks in front of the user.
What's next for Walkthru
Getting the landlord to co-sign at move-in, so both sides share one record. A live, voice-guided walkthrough using the Gemini Live API. Talking to university off-campus housing offices about offering it to students. And more states and languages.
Built With
- cloud-build
- cloud-storage
- ffmpeg
- firebase
- firebase-auth
- firestore
- gemini
- gemini-api
- google-calendar
- google-cloud-run
- google-genai-sdk
- google-maps
- maps-javascript-api
- next.js
- places-api
- react
- shadcn-ui
- tailwindcss
- typescript