instagram_personalization

Inspiration

The disconnect between the incredibly rich taste signals embedded in our social media feeds and the sterile, generic experience of modern shopping websites. When users scroll Instagram, they curate deep, nuanced aesthetic profiles—but when they shop online, retail algorithms start from zero. We wanted to build an AI pipeline that bridges this gap, translating casual social media browsing into highly personalized, algorithmically curated shopping feeds without compromising privacy.

What it does

Phia is an autonomous personalization engine built in two parts: a Chrome extension and a companion Next.js web application.

The browser extension acts as a secure, local reader of a user's Instagram browsing (extracting handles, captions, and posts from the DOM). It captures both explicit metadata (keywords, followed brands) and raw visual data (dominant color palettes, brightness, saturation bands) via offscreen canvas pixel extraction.

This data forms a unified InstagramTasteProfile synced securely to a Supabase backend. The web application then ingests this real-time graph, mapping it against a live catalog of thousands of scraped retail products, dynamically re-ranking the entire storefront visually and textually to perfectly align with the user's revealed aesthetic preferences.

How we built it

• Chrome Extension API: Background workers and content scripts manage local data extraction seamlessly as the user browses Instagram, preventing the need for official API keys or OAuth setups.
• Supabase (PostgreSQL): Serves as our real-time database and cross-platform source of truth, synchronizing the taste state between the extension writer and the web app reader.
• Next.js & React: The web frontend housing the live shopping grid, leveraging extensive useMemo hooks for high-performance virtual rendering.
• Vercel & Edge Functions: Used to build a universal serverless image proxy. This bypassed strict CDN blocks, allowing us to safely feed remote images into local <canvas> elements for pixel-level color reading.
• Custom Ranking Heuristics: Vanilla algorithms for visual similarities—scoring products by brightness deltas, saturation bands, and palette similarity, combined with heavily weighted keyword match scoring.

Challenges we ran into

• Cross-Origin Read Blocking (CORS): When attempting to extract visual palettes from third-party retailer CDNs (like Talbots or Shopify) using offscreen canvases, strict CORS effectively halted the process. We overcame this by launching a dedicated universal image proxy deployed on Vercel to explicitly handle pixel-level canvas reading, whilst ensuring the UI continued relying on direct <img> fetching to preserve speed.
• State Synchronization & Performance: Attempting to synchronize asynchronous personalization descriptors across a heavy grid of thousands of retail items resulted in crippling React re-render loops. We resolved this via architectural refactoring, isolating data flows, and implementing strict useRef caching mechanisms.
• Algorithmic Imbalance: During integration, our ranking script over-rewarded explicit, rigid text matches (e.g., throwing any product labeled "sporty" to the top) while punishing incredibly accurate but untagged visual matches. We had to perform live algorithmic tuning to heavily invert our multipliers, forcing nuanced visual aesthetics to dominate raw keyword matches.

Accomplishments that we're proud of

We successfully built a proxy that leverages social graphs without requiring the user to compromise account credentials or authorize invasive APIs. Furthermore, we achieved Zero-Storage Visual Profiling: Phia never downloads, saves, or screenshots actual Instagram photos. Images are loaded ephemerally into invisible DOM canvases, mathematically abstracted into pixel hex profiles (brightness, dominant hues), and then immediately discarded. We only sync the math—not the photos—ensuring total user privacy and massively reduced database costs.

What we learned

Tuning consumer-facing recommendation systems requires an intimate, delicate balance between literal classification (keywords like "luxury" or "streetwear") and abstract visual mapping (brightness deltas, dominant RGB patterns). Abstracting shopping to a color and vibe dimension almost universally provides a better user experience than rigid textual taxonomy.

What's next for instagram_personalization

• Edge-Based Computer Vision: Moving beyond color maps to integrate lightweight, local machine learning models directly into the extension. We want to classify localized product cuts, silhouettes, and fits (e.g., explicitly recognizing a dropped shoulder vs. a tight crop) on the fly.
• Platform Agnosticism: Generalizing the DOM scrapers to seamlessly ingest taste signals across Pinterest, TikTok, and direct fashion blogs.
• Real-time Price Engine Expansion: Deepening the Deal Watch infrastructure to reliably scrape resale marketplaces for identical styles recommended by the algorithm.

Built With

• chrome
• css
• html
• next.js
• node.js
• postgresql
• react
• supabase
• typescript
• vercel