JwelLens

Inspiration Jewellery shopping is often based on visual preference, but traditional search mostly depends on keywords. If someone sees a jewellery design they like but doesn't know the exact name or terminology, finding a similar product can be difficult.

We wanted to make jewellery discovery more natural: users should be able to describe, show, speak, or sketch what they are looking for.

That idea led us to build JeweLens, a multimodal AI jewellery retrieval system that turns a user's intent into relevant jewellery results.

What it does

JeweLens allows users to search for jewellery using multiple input modes:

Text — describe the jewellery you want in natural language. Image — upload an existing jewellery image and find visually similar designs. Voice — describe what you're looking for naturally through speech. Sketch — draw a rough jewellery design and retrieve similar products. Category filtering — narrow results by jewellery type such as rings or necklaces. Wishlist & Cart — save interesting designs and manage selected products.

Instead of matching only exact keywords, JeweLens focuses on semantic and visual similarity, making jewellery discovery more flexible and intuitive.

How we built it

We designed JeweLens as a full-stack AI application.

The frontend was built using React, while the backend was developed with FastAPI.

For multimodal retrieval, we used CLIP embeddings to represent text and jewellery images in a shared embedding space. This allows a text description and an image to be compared based on their semantic similarity.

The retrieval pipeline works roughly like this: \text{User Input} \rightarrow \text{Embedding} \rightarrow \text{FAISS Search} \rightarrow \text{Filtering} \rightarrow \text{Results}

We used FAISS for efficient vector similarity search across the jewellery dataset.

For voice input, speech is converted into text before entering the retrieval pipeline, while sketches and images are processed as visual queries.

The overall architecture separates the React user interface, FastAPI backend, and AI retrieval layer, making the system easier to extend.

Challenges we ran into

One of our biggest challenges was retrieval relevance.

A jewellery item can be visually similar while still being the wrong type. For example, a necklace might share similar colours and patterns with a ring, but returning it for a ring-specific query would not be useful.

We therefore had to combine semantic similarity with category-based filtering instead of relying purely on vector similarity.

Another challenge was supporting different input types while keeping the experience consistent. Text, images, voice, and sketches all produce different forms of input, so designing a common retrieval pipeline required careful preprocessing and integration.

We also had to balance retrieval quality and speed, since searching through large numbers of embeddings needs to remain responsive for users.

Accomplishments that we're proud of

We are proud that JeweLens goes beyond a conventional jewellery search interface and demonstrates how multimodal AI can change the way users discover products.

Our key accomplishments include:

Built a working multimodal jewellery retrieval system.
Integrated CLIP embeddings for semantic and visual search.
Implemented FAISS vector similarity search.
Supported text, image, voice, and sketch-based discovery.
Added category-aware retrieval to improve relevance.
Built a complete React + FastAPI application rather than only a model prototype.
Created an experience where users can search based on their intent rather than exact product terminology.