Nali | Devpost

Inspiration

We came into this hackathon unsure of what to cook up, but SAP's challenge resonated with our innovative minds and made us want to build a system that solves traditional Lost and Found problems: logistics, data privacy, and manual item matching.

What it does

Our solution is an intelligent, privacy-first lost and found platform which acts as a middleman between users and assistants. The core of iNauly is its AI matching engine, which cross-references text or image user inquiries against the system's private inventory.

How we built it

We separated the system into 4 components:

The frontend is built with JS and HTML, and directly interacts with our database.
The AI Engine is the core of the multimodal (text-to-image and image-to-image) matching algorithm using Openai's CLIP model
The Weaviate Vector database which stores both the users/inquiries/input images and the lost items images. +1 added feature: Openai DALL-E 3 used to generate images from prompts, for users without a picture on hand

Challenges we ran into

Components integration: integrating the stack together while some of us were trying them for the first time required a lot of testing and debugging to make sure that the system functioned as a whole.
Making our solution interesting: The largest challenge we ran into was selecting which features to implement from a space full of potential features that could make our solution stand out from the competition. Ultimately, we settled on Image generation and Vectorization.

Accomplishments that we're proud of

Successful first-time use and integration of multi-modal matching
Complete database placed in Weaviate
Identifying and discussing important tradeoffs for the item comparison method and database

What we learned

Brainstorming AI solutions for item matching allowed us to apply Linear algebra theory to existing SOTA solutions

What's next for Nali

Ownership verification: we want to implement a final verification step where the system quizzes the user about a specific and unique characteristic of their item (e.g., "What is the wallpaper on the lock screen?").
Smart notifications: since items can be found after users have submitted inquiries about them, smart notifications can be implemented to alert users the instant a potential match is found (efficient reverse search FOUND items to LOST items).