SnapMend

## Inspiration

Every homeowner knows the feeling: something breaks, you stare at it, and you have no idea where to start. A crack in the drywall, a cabinet hinge that won't stay, a leaking pipe fitting — these problems look deceptively simple but hide a dozen wrong turns. We wanted to build something that could look at the problem with you and give you a real, actionable plan in under a minute.

## What it does

SnapMend is an AI-powered home repair assistant. You take a photo of the broken thing, describe what you're seeing — by typing, by voice, or both — and SnapMend produces a complete repair plan:

Diagnosis — what's actually wrong, backed by evidence from the photo and your description
Safety warnings — what to watch out for before you pick up a tool
Step-by-step instructions — practical, homeowner-level guidance
Materials list & cost estimate — so you know what to buy and roughly what to spend
Product recommendations — real purchase links from reputable hardware stores, found via live web search

Every analysis is saved to a history screen so you can come back to it later.

## How we built it

The backend is a NestJS API that orchestrates three OpenAI calls in sequence:

Transcription — if a voice note was uploaded, gpt-4o-mini-transcribe converts it to text
Repair plan generation — gpt-4.1-mini receives the image (as a base64-encoded vision input), the text description, and the transcript, then returns a structured JSON repair plan enforced by the OpenAI Responses API's json_schema format with strict: true
Product search — gpt-5 with the web_search tool finds real product URLs matching the diagnosed repair, returning structured recommendations in the same strict-schema format

The frontend is React 18 + TypeScript + Vite, using React Router for client-side navigation and the browser's MediaRecorder API for in-browser voice capture. Everything is served same-origin from NestJS's static asset middleware, so there's no separate frontend server. The whole stack runs in a single Docker container via docker compose.

## Challenges we ran into

OpenAI structured outputs and optional fields — OpenAI's strict: true mode requires every property in a JSON Schema object to appear in the required array. There's no such thing as a truly optional field; instead you union the type with null. Debugging this took longer than expected because the error message ('required' is required to be supplied and to be an array including every key in properties) is easy to miss when it arrives as a 400 mid-pipeline.

Pipeline latency — chaining transcription → vision analysis → web search means the request can take 30–60 seconds. We added a withTimeout wrapper on the product search step so a slow web-search response degrades gracefully (returning an empty recommendations list) rather than hanging the whole request.

Voice recording in-browser — MediaRecorder MIME type support varies across browsers. We detect whether audio/webm is supported and fall back to audio/mp4, then send the blob as recording.webm regardless so the backend and OpenAI transcription endpoint always get a consistent filename.

## What we learned

The OpenAI Responses API's structured output mode is powerful but strict — schema design matters as much as prompt design
Graceful degradation at each pipeline step (timeout + fallback) is essential when chaining multiple AI calls
Serving a full-stack app from a single NestJS container with Vite's production build keeps deployment dead simple

## What's next for SnapMend