Palette: Personal AI Color Analysis

Inspiration

Seasonal color analysis blew up over the past year, but actually getting it done still means booking a consultant and paying somewhere between $100 and $300 for someone to tell you which colors work on you. And even after all that, you're on your own when it comes to buying anything. I kept coming back to one thought: almost all of that could come from a single selfie.

I wanted anyone to be able to figure out their colors at home in a few seconds, and then immediately see real products that match, so they'd know exactly what to look for before they ever set foot in a store.

There's a big shopping angle too. Walking into a store you've never been to is basically a guessing game about what's going to look good on you, and that guesswork is one of the quiet reasons so much of what people buy ends up getting returned. Palette takes the guessing out of it by matching you to items the store actually carries. That's a win for the shopper and a win for the store, because people buy more, and send less back, when what they picked actually suits them.

What it does

You upload or snap a photo. It runs smoothly on a phone and can be added to your home screen like a regular app. A vision model reads your skin undertone, hair, and eye color and builds a personal palette, usually 5 to 7 colors, each with its own hex code and name. From there it searches a product catalog and pulls up real makeup, clothing, and skincare matched to your palette, complete with prices, images, and links. Use it at home before you shop, or pull it up on your phone while you're standing in a store you've never been to. Built for anyone, no fashion background required.

How we built it

The whole thing is one Next.js app hosted on Vercel, and the entire backend lives inside a single serverless function. Cheap to run and easy to scale without any new infrastructure.

Selfie -> /api/analyze (serverless) -> Qwen2.5-VL-72B (vision) -> strict JSON palette -> MiniLM embedding -> Pinecone similarity search -> matched products

• Vision: Qwen2.5-VL-72B through Hugging Face, prompted to output strict JSON, with a parser that cleans up messy responses and falls back safely if something breaks.

• Matching: the palette gets turned into a text query, embedded with all-MiniLM-L6-v2, and compared against product vectors in Pinecone using cosine similarity.

• Catalog: pulled from Google Shopping via SerpApi, embedded, and stored in Pinecone. Re-running the pipeline grows or swaps out the catalog. Because the API is stateless and the catalog is just a vector index, the exact same setup handles one person or a thousand, and a store could drop in their own inventory just by swapping the index. Scaling it is a configuration change, not a rebuild.

Challenges we ran into

Getting the model to return clean JSON every single time was probably the hardest part. Vision models really like to wrap answers in code fences or tack on an extra sentence of explanation, so I wrote a strict prompt with an exact template, then backed it with a parser that strips the junk, checks every field, and falls back to a safe default palette if anything looks off. The app never breaks, even when the model misbehaves.

Matching was its own headache. A palette color named "Twilight Blue" doesn't automatically line up with a product tagged "navy," so I pull a basic color word out of each hex code and let the embeddings bridge the gap between palette colors and how products are labeled.

Hosting threw a real curveball. The smaller vision model I started with went down, and the free container hosting I'd planned on moved behind a paywall right when I needed it. So I re-architected on the fly, folding the entire backend into the Vercel function and switching to a hosted embeddings endpoint. Before trusting the switch, I verified the hosted vectors matched what I was getting locally, down to a cosine similarity of 1.0, which meant I didn't have to touch the matching logic. Everything still runs on free tiers with hard limits, so nothing can quietly rack up a bill, and it's deployed and genuinely usable today, with plenty of room to build on.

Accomplishments I am proud of

Getting from a single selfie to five to seven matched colors and shoppable products, entirely through one serverless function, with no dedicated backend server anywhere. The fallback logic held up even when the model returned complete garbage, which happened more than I expected during testing. And confirming the hosted embeddings matched my local ones exactly, before committing to the switch, saved me from rebuilding the entire matching layer under a deadline.

What I learned

I picked up a lot about pulling reliable structured output from a language model that really does not want to give it to you, building vector search from the ground up, running a large vision model without paying for GPU hosting, and turning a single serverless deploy into something that installs and works like a real mobile app.

What's next for Palette: Personal AI Color Analysis

This is really just the starting point. A few directions worth taking it:

• Matching clothing to specific occasions, like interviews, weddings, work, or a night out, so it goes from "these are your colors" to "here's exactly what to wear for this."
• Kiosks or tablets in makeup and clothing stores, where someone scans their face and gets matched to what's actually in stock, so buyers can shop straight to what suits them instead of grinding through trial and error.
• A lightweight "find my colors" widget that brands could drop into their own site, so customers buy the right thing the first time.
• Per-store catalogs, saved palettes and accounts, stronger coverage across all skin tones, and tighter matching across the board. All of it runs on the same core pipeline, so scaling this from a demo to something a real store could put in front of customers is mostly about adding to it, not rebuilding it.

Built With

• all-minilm-l6-v2
• computer-vision
• embeddings
• fastapi
• google-shopping
• hugging-face
• javascript
• next.js
• node.js
• pinecone
• pwa
• python
• qwen2.5-vl
• rag
• react
• sentence-transformers
• serpapi
• serverless
• vector-search
• vercel
• vision-language-model