FoundAI

Inspiration

We all have experienced the common frustration of losing items on campus and the inefficiency of existing lost-and-found systems. We realized that we needed a faster and more accurate way for students and faculty to find their lost belongings, and we wanted to bring that platform via a simple website that's not only easy to use, but also solves these grueling problems.

What it does

FoundAI vows to solve this problem by utilizing PyTorch image processing/classification to help you and others find what they're looking for. Simply upload a picture of your lost item to FoundAI and it will automatically give you results for similar items that people have found across your campus. Don't have a picture? You can instead effortlessly describe your lost item to our search bar and using object detection through images, our AI can easily match items that resemble the found items.

How we built it

We used React and Javascript for our frontend. For our backend, we used Python, Flask, MongoDB, and AWS S3. In our frontend, we would call REST API calls to interact with our MongoDB cluster to upload lost and found information. We would upload the photo to the AWS S3 bucket and then update the document with the S3 link. For our search functionality, we utilized MongoDB search indexes and PyTorch. For MongoDB search indexes, we would take the user's search query and then evaluate potential matches. For PyTorch, we used a ResNet18 model to evaluate images and match the user's search photo to a potential match. This let the user quickly either find a matching found/lost item.

Challenges we ran into

We faced several technical challenges throughout the development of FoundAI. Initially, none of us were familiar with using PyTorch, MongoDB, or hosting images on AWS S3, which meant a steep learning curve as we tried to integrate all these technologies into our system. Another challenge was determining an optimal threshold for the image classification similarity rank in PyTorch. We needed to fine-tune this parameter to ensure the system returned accurate matches for users searching for their lost items. Additionally, while we used the ImageNet database to train our model, we encountered difficulties due to the overwhelming number of irrelevant keywords. We had to filter out unnecessary keywords and keep only the top relevant ones, which improved both the performance and accuracy of our search algorithm. Balancing all of this with the integration of the React frontend added complexity to the project.

Accomplishments that we're proud of

We were able to fine-tune the image classification model to accurately rank similar images based on user uploads. Overcoming the challenge of filtering ImageNet data to improve search accuracy was a significant achievement, as it greatly enhanced the user experience. Finally, creating a scalable and user-friendly platform that connects lost items to their owners on campus is something we're particularly proud of.

What we learned

We learned how to efficiently manage and process image data using PyTorch, as well as how to store and retrieve large datasets using MongoDB. Working with AWS S3 taught us the ins and outs of hosting and serving images, and building the frontend with React helped us hone our full-stack development skills. Additionally, we gained a deep understanding of how to fine-tune machine learning models for practical applications, and how to optimize search algorithms by filtering out unnecessary keywords.

What's next for FoundAI

In the future, we plan to incorporate augmented reality (AR) to allow users to visualize where their lost items might be in real-time based on other users' reports. We also want to improve the accuracy and speed of our image classification model and expand the platform to support multiple campuses and institutions.