Snap&Go - Improve 311 using Generative AI

Screen1
Screen2

Inspiration: We created Snap&Go out of personal need. As busy working parents and immigrants, we know how tough it can be to find time to report issues in our neighborhood. We needed an app that understands our fast-paced lives and speaks our language—literally. Snap&Go is that friendly neighbor, making it easy for all of us to help keep our community a great place to live, one click at a time.

What it does : Snap&Go reinvents how we report community issues by making it incredibly easy. Just one click - that's all it takes to reach city services, thanks to our Generative AI that cuts through the red tape of forms and language barriers. Not only does it make reporting hassle-free for everyone, but it's also smart enough to ensure your concerns are sent to the right place right away, significantly speeding up the response time. With Snap&Go, you get swift action on issues that matter to you, in an app that's as inclusive as it is efficient.

How we built it :

We built Snap&Go by integrating two Large Language Models (LLMs), including miniGPT-4, to transcribe images, and the more powerful GPT-4 to classify issues accurately. For a smooth user experience, we developed the web interface with React, supported by the versatility of JavaScript.

Challenges we ran into :

We faced a few hurdles, like combining different AI models for image and text processing, which was quite complex. Then, we had to refine our AI prompts to prevent errors in understanding the issues reported. Lastly, keeping our app fast meant we had to limit image sizes to manage the API's response times effectively.

Accomplishments that we're proud of :

We're especially proud of achieving the image-to-text translation and integrating it seamlessly to autofill report forms—an intricate feat. Moreover, implementing multilingual support quickly and efficiently to cater to diverse users was a significant triumph for our team.

What we learned :

Hallucinations are a problem with LLM models but a structured prompt-engineering can provide some solutions and we can do wonders with the new GenAI tech.