Inspiration
Around 300 million people live with colour vision deficiency (CVD)[cite: 1, 4]. More than half are mild deuteranomals who go undiagnosed and rely daily on others to verify basic styling choices—from identifying skin redness to telling navy blue from dark purple[cite: 1, 4].
Traditional colour wheels and digital simulations fail because our user's eyes already filter what is on screen[cite: 1, 4]. They need sensory substitution[cite: 1, 4]: an objective, calibrated numerical measurement of their face and clothes, backed by an application that explicitly refuses to answer when lighting conditions would yield a false reading[cite: 1, 4].
What it does
KROMA starts empty on purpose—every colour on screen is measured from the user's real photos on their device[cite: 1, 4]:
- Vision Calibration (F1): Derives exact deficiency type and severity via a Farnsworth D-15 arrangement test without asking users to self-diagnose[cite: 4].
- Personal Colour Portrait (F2–F5): Extracts skin, hair, eyes, lips, Fitzpatrick phototype, and baseline redness scores using YouCam Skin AI with lighting normalisation[cite: 4].
- Audited Wardrobe & Colour Families (F6–F11): Isolates clothes with
/task/sod, extracts dominant colours (filtering out fold shadows), warns against duplicate purchases ("you already own this colour"), and groups garments into distinguishable colour families[cite: 1, 4]. - "Get Dressed" Outfit Generator (F19): Automatically builds clash-free outfits matched against the user's personal complexion and hair palette, eliminating morning guesswork[cite: 3, 4].
- Retail Assistant & Try-On (F12–F14): In-store shoppers photograph garments to get an instant wardrobe compatibility percentage and VTO renders via
/task/cloth-v4[cite: 1, 3, 4]. - Closed-Loop Makeup Studio (F15–F16): Recommends complementary correctors for facial redness, applies them virtually via MCP, and re-measures the user's real face to mathematically verify redness reduction[cite: 3, 4].
- Assisted Contrast (F18): Re-injects lost contrast into visible channels via LMS daltonisation so colliding colours become distinguishable without altering raw measurement data[cite: 1, 4].
How we built it
- Stack & Architecture: Next.js 15 (App Router), TypeScript (Strict), Tailwind CSS v4, and mobile-first PWA deployed on Vercel[cite: 1, 4].
- Local-First Privacy: All user profiles, wardrobes, and images persist in client-side IndexedDB (
idb) with zero external database logins[cite: 1, 4]. - Colour Science: CIELAB/CIEDE2000 metrics via
culori, paired with offline-verified Machado (2009) simulation matrices[cite: 1, 4]. - Server-Side Image Processing: Built with
sharpto guarantee EXIF rotation, alpha-to-white flattening, and 4:4:4 chroma subsampling preservation[cite: 1, 4]. - YouCam API & MCP: Server-side Route Handlers integrate 8 endpoints across REST and Model Context Protocol (MCP) servers (
beauty,fashion,creators)[cite: 3, 4].
Challenges we ran into
- Lighting Normalisation on Fabrics: While
/task/lightingremoves colour casts on portraits (ΔE 0.5–0.7), it distorted flat fabric (shifting a daylight shot from ΔE 1.8 to ΔE 20.2 away from truth)[cite: 1, 4]. We isolated/task/lightingstrictly to faces, using local pixel metrics for garments[cite: 1, 4]. - Shadows vs. Patterns: Lightness clustering mistook fabric fold shadows for multi-coloured patterns[cite: 1, 4]. Clustering strictly by hue angle and splitting neutrals only at histogram valleys took a misreported orange t-shirt from "37% dominant / patterned" to 99% solid[cite: 1, 4].
- REST vs. MCP Parity: The same image returned skin tone within ΔE 1.0, but iris colour diverged by ΔE 6.4[cite: 1, 4]. We designated REST as the single source of truth for persistent measurements, using MCP for multi-server workflow orchestration[cite: 1, 4].
Accomplishments that we're proud of
- Brutal Honesty as a Feature: Every measurement renders an explicit confidence level;
"refuse"is an intentional product state, not an error[cite: 1, 4]. - Zero Generative Hallucinations: Language models are banned from deciding colour names, numbers, or clash verdicts—colour naming is 100% deterministic[cite: 1, 4].
- Rescuing Features with MCP: When
/task/makeupreturned 404 on REST, we usedAI-Makeup-Virtual-Try-Onon thebeautyMCP server with arbitrary hex inputs to complete the closed-loop redness verification[cite: 3, 4]. - Undocumented Capability Discovery: By inspecting MCP
tools/list, we discovered the enumouter(resolving an HTTP 400 error caused by the documentedouterwear), unlocking reliable jacket/coat try-ons across both transports[cite: 3, 4].
What we learned
When building accessibility software for users who cannot visually audit the output, honesty is an engineering constraint[cite: 1, 4]. Every calculation must be mathematically provable, deterministic, and verified by synthetic test harnesses[cite: 1, 4].
What's next for KROMA
- Implementing Brettel (1997) two-half-plane projection specifically for tritanopia[cite: 4].
- Calibrating phototype-specific baseline redness thresholds across all Fitzpatrick scale phototypes[cite: 1, 4].
- Exposing KROMA's deterministic colour engine as an MCP server for external AI styling agents[cite: 3, 4].
Built With
- accessibility
- color-blindness
- culori
- fashion-tech
- indexeddb
- model-context-protocol
- next.js
- sharp
- tailwindcss
- typescript
- vercel
- youcam-api