ReFoundSG

Inspiration

What started as a meme turned into a real pain. Most, if not everyone, has been in the situation where you lose an item and the panic sets in: Where have I lost my item? How am I going to retrieve it? Currently, there exists telegram channels, methods of retrieval like SPF's lost and found system or NUS' Lost & Found system. However, all these are a hassle, having to scroll through a fragmeneted mess of Telegram messages, or filling up multiple forms, forcing victims to actively recall the situation, all of which can definitely be prevented. Hence, we asked ourselves: "How might we stop the doom-scrolling and make the items find the owners?"

What it does

ReFound is a telegram-based image matching engine that reduces the search process and turns it into an active notification system. With user-friendly interfaces as well as easy to follow steps, ReFound makes the Lost & Found process easier, and a report can be submitted in 1 minute, having matches found under 1 minute. Once a match is obtained, users can chat anonymously with one another which ensures privacy and makes the retrieval process easier.

For Finders: Pin point the exact location using precise location finder of where the item was found, snap a photo and upload it to the telegram bot. For Owners: Describe what you lost. The magic: Our bot instantly matches the visual features of the found item with the semantic description of the lost item, using a confidence score, owners can scroll through various matches that matches similar descriptives features and contact the finder should the match be positive.

How we built it

We prioritised speed and fricitionless UX. We did not wantusers to download a new app hence we built ReFound as a telegram bot since many people use Telegram, which makes the process more user-friendly since users are confident with an environment familiar to them. Interface: Python-telegram-bot for a conversational, intuitive UI. Python backend handling state management and conversation flows.

Challenges we ran into

The "Language Gap": A finder sees a "white plastic case" but the owner lost his "Airpods Pro 2nd Gen". Simple text search fails here. Implementing Multimodal Vector search allowed us to bridge this gap which matches the visual reality to the semantic concept Privacy & Safety: With scams becoming more prevalent, we were afraid that people might abuse such a system. Hence we implemented an anonymous chatting system which ensures that users do not leak information to one another. In the future, we plan to operate on a centralised drop-off location system, which minimalises face-to-face contact between finders and owners.

What we learned

Making a socially advantageous system, building a bot that lives inside existing community chats makes it a fast way to solve the problem of two-sided marketplaces. We also learnt that certain daily problems can be solved as long as we put our minds to it.

What's next for ReFoundSG

Integrating a centralised drop-off location for finders to drop the item off and for owners to easily pick up items at their own time, without the hassle of face-to-face meetups, which improves security and anonymity.