What The Fit

Inspiration

You're scrolling Instagram or TikTok, you spot an outfit someone's wearing, and you think "I need that jacket." Then you spend twenty minutes reverse-image-searching, reading comments, and giving up. Every piece of good content on social media doubles as an accidental ad for clothes nobody can actually buy. I wanted to close that loop.

What it does

What the Fit is an iOS app with a native share extension. From inside Instagram, TikTok, or YouTube, you hit the share button, pick "What the Fit," and that's it. About 30 seconds later you get every person in the clip, every piece of clothing they're wearing, and a real "Buy Now" link with price and product image for each item.

How it's built

iOS share target: React Native + Expo (SDK 52) with expo-share-extension. The extension captures the URL from Instagram/TikTok/YouTube, submits a job to the backend, and hands off to the main app for progressive results.
Video fetch: yt-dlp in a self-managed AWS Lambda layer that rebuilds with the latest binary on every deploy. Instagram has a 6-method fallback chain (yt-dlp with the share-tokenized URL → canonical URL → mobile API → embed → GraphQL → og-meta) so one failed method doesn't kill the job.
Pipeline (processor Lambda, ~30s wall time): the processor downloads and normalizes the video first, capping at 720p progressive MP4 (roughly 3× smaller than the default 1080p DASH variant and a meaningful latency win). Then it samples 14 evenly-spaced frames with ffmpeg and runs AWS Rekognition face detection on all of them in parallel. Candidate faces get indexed into a scratch collection, and a pairwise SearchFaces pass builds a similarity graph that we union-find into identity clusters, with a post-merge step to catch the same person read as two clusters across big angle changes. For each cluster we extract a body crop and send it to AWS Bedrock Nova Lite, which names every visible clothing item with category, color, brand guess, and a short description. Finally, per item, we fan out to SerpAPI Google Shopping for candidates and then hit SerpAPI's Immersive Product endpoint for the best match, filtering out resale hosts so the "Buy Now" link lands on a real retailer.
Infrastructure: AWS SAM - Lambda, DynamoDB (jobs), SQS (processor queue), S3 (video staging). HTTP API with optional Cognito auth. - Progressive results: the processor publishes 4 checkpoints (videoUrl → sampling thumbnails → person crops → enriched items), so the phone can render each stage the moment it's ready rather than sitting on a 30-second spinner.

What is Next

Publishing on the app store
Connecting with more social media platforms (X, Reddit, YouTube shorts, LinkedIn profiles)
Finding something cheaper and faster than SerpAPI for finding products online