MuseLens — Multimodal Design Assistant

Designers need fast, relevant visual inspiration and rationale. Nova Lite can be used for multimodal embeddings for image search and for critique/brief generation.

Comment

Inspiration

We built MuseLens — Multimodal Design Assistant because creative discovery is inefficient: designers spend hours chasing visual references, iterating color palettes, and rationalizing decisions that should be scaffolded by a smart assistant. We were inspired by the recent advances in multimodal embeddings and foundation models (notably Amazon Nova) which make it possible to semantically link images, text, and UI screenshots at scale. MuseLens asks a simple question: what if a tool could surface not just visually similar assets, but explain why they work and suggest concrete next steps for layout, color, and accessibility?

What it does

MuseLens helps designers move from idea → iteration faster by combining visual search, clustering, and model-driven critique. Key user flows:

  • Ingest: upload images (moodboards, screenshots), paste text briefs, or link a Figma frame.

  • Discover: returns ranked, multimodal inspirations (images, micro-briefs, palette suggestions) using embedding nearest-neighbors.

  • Explain: for each suggestion the assistant gives a short rationale (composition, hierarchy, color contrast, accessibility hints) created by a reasoning model.

  • Synthesize: auto-generates 2–3 layout variations and color palettes with explicit tokens you can copy into Figma or CSS.

  • Iterate: you can ask the assistant to “make it friendlier / more corporate / high-contrast” and it performs a targeted re-ranking + new suggestions.

Example of the scoring function we used for retrieval (simplified):

score(q, d) = α ⋅ cosine(e_q​, e_d​) + β ⋅ recency(d) + γ ⋅ studio_vote(d)

where:

cosine(u, v) = (u · v) / (||u|| ||v||)

and

α + β + γ = 1


How we built it

High-level architecture (components and their responsibilities):

  1. Frontend

    • React-based UI for uploads, quick previews, and the iteration panel.
    • Lightweight Figma plugin (prototype) to push or pull frames.

  2. Embedding & Retrieval

    • Multimodal encoder to turn images and text into vectors.
    • ANN index (FAISS / HNSW) for fast nearest-neighbor retrieval.
    • Vector post-processing: PCA-whitening and L2-normalization before indexing.

  3. Reasoning & Critique

    • A reasoning model (Nova-class lite model in our prototype) consumes a small context window: {query, nearest examples, brief, accessibility rules} and emits a structured critique and actionable outputs (palette hexes, layout specs, CSS tokens).

  4. Synthesis

    • Template-driven prompt templates generate: palette palettes, typographic scale, and 2–3 layout wireframes.
    • Optional style-transfer / image augmentation module for rapid visual mockups.

  5. Orchestration & Storage

    • Node.js backend orchestrates embedding calls, retrieval, and reasoning prompts.
    • S3 for assets; a small DB for sessions and user annotations.

Selected technical details and equations

  • We cluster retrieved inspirations to present diverse modes. We used k-means with the usual objective. To minimize:

    min_{C_1, ..., C_k}  Σ(i = 1 to k)  Σ(x ∈ C_i) || x − μ_i ||²

    where:
    C_i = cluster i
    k = number of clusters
    x = data point
    μ_i = centroid of cluster C_i
    ||x − μ_i||² = squared Euclidean distance between point and centroid

  • To balance precision vs. serendipity we mix top-(k) by similarity with a stochastic sampling step (softmax with temperature (T)):

    P(d ∣ q) ∝ exp(score(q, d) / T)


Challenges we ran into

  • Subjectivity of “good”: aesthetic preference is heavily subjective. Metrics like MRR or NDCG only partially capture usefulness. We relied on qualitative sessions alongside retrieval metrics.

  • Dataset curation & licensing: finding high-quality, license-safe images that span UI, print, and product design required manual curation and clear attribution metadata.

  • Embedding alignment: aligning image and text spaces so that a brief like “playful, geometric, pastel” returns usable visuals took many iterations of prompt + normalization tweaks.

  • Latency vs. quality: high-dimensional retrieval and multi-step reasoning produce latency. We had to add caching, smaller “fast” models for initial results, and queue higher-quality passes.

  • Interface friction: integrating suggestions into designers’ existing tools (Figma, Slack, Notion) required careful UX—export formats and copyable tokens matter more than the quality of the suggestion itself.

Accomplishments that we're proud of

  • Delivered a working end-to-end prototype: upload → retrieve → critique → synthesize, with exportable palettes and layout tokens.

  • Built a small Figma connector that converts MuseLens palettes and tokens into editable styles inside Figma.

  • Designed compact prompt templates and an embedding post-processing pipeline that improved perceived relevance during our pilot.

  • Ran an informal pilot with 5 designers (volunteer testing) to validate flow and capture pain points—notes drove the next roadmap items.

What we learned

  • Prompt + retrieval are a package: good prompts can’t fully compensate for weak retrieval; conversely, great retrieval with poor prompting is confusing. They must be tuned jointly.

  • Serendipity is a feature: designers value unexpected pairings — perfect nearest-neighbor accuracy isn’t always the goal. We therefore deliberately introduce a controlled amount of diversity using sampling temperature.

  • Small UX details matter: keyboard shortcuts, clipboard-ready CSS tokens, and one-click export to Figma increased adoption more than improving model accuracy by a small percent.

  • Practical evaluation: combine quantitative metrics (MRR, recall@k) with short, task-based user evaluations (time-to-first-usable-mockup).

What's next

Short- and mid-term roadmap items:

  1. Figma-first integration — make MuseLens a first-class plugin: live suggestions, one-click style sync, and automatic token generation for design systems.

  2. Personalization — user profiles that bias retrieval toward their preferred visual language (learned from saved assets).

  3. Accessibility-first mode — generate palette and contrast-aware suggestions that conform to WCAG thresholds automatically. We’ll enforce contrast ratio for normal text: $$L = {(L_1 + 0.05) / (L_2 + 0.05)} >= 4.5$$
  4. Evaluation framework — formal A/B tests with larger designer cohorts to measure time-savings and preference lift. Use metrics like time-to-first-usable-design and subjective likability scores.

  5. Production optimizations — quantize embeddings, add per-user caching, and move heavy reasoning into asynchronous batch passes while keeping a fast-first UX.

Built With

Share this project:

Updates

Submission history