Instruct IQ

Our project aids elders facing technical hurdles by using image capture and processing to get the next steps. Leveraging Language Models (LLM), we give users with a sequence of troubleshooting steps.

Comment

The inspiration

My inspiration stems from my parents, who struggle with understanding various features of Facebook. Explaining the icons or where to click or type is challenging due to her educational background. Our annotated images aim to provide direct guidance to help individuals like her navigate technology with ease. This motivation drives us to build a product that empowers seniors worldwide, ensuring they don't feel disappointed or left behind.

Idea Video: https://www.videoscribe.co/app/preview/d8d65832-5b52-4460-be68-f872a3f1eede/ PlayList: https://www.youtube.com/playlist?list=PLSPImV-J0zlmwSoTFdlD0puZTUZWY18cO

Our target audience

  • Our project aids elders facing technical hurdles by utilizing image capture and processing to ascertain the next steps.
  • We would also want to target companies willing to make their products more accessible.

  • Leveraging Language Model (LLM) technology, we present users with a sequence of troubleshooting steps.

  • Examples include resetting Facebook accounts, recovering passwords, or managing daily deliveries. The domain aims to push the boundaries of automation while maximizing feedback from seniors to optimize usability.

Technologies Used

We build our backend as worker role using Intel Developer Cloud and Intel Extension for Transformer libraries backed by Chroma DB as a workflow running server requests to process LLM queries and RAG. The front-end is a Reflex app built purely in Python. The app serves as a web role with an image processing backend to annotate images uploaded via camera or as a file.

Challenges we ran into

This was our first hackathon using LLM, so getting the terminologies right presented a significant challenge. The team at Intel provided us with some amazing sessions, explaining their tech stack and clarifying our doubts along the way. Additionally, we had to use Reflex for our front-end, which was relatively new and required a different learning curve compared to someone with a JavaScript background. The team was incredibly helpful in providing continued support, and we thoroughly enjoyed experimenting with their platform and exploring its boundaries.

Accomplishments that we're proud of

Over the course of 36 hours, we dedicated a significant amount of time to gathering feedback from mentors and sponsors regarding our pitch. This marked our first accomplishment, as it provided validation for our idea. We were successful in crafting an acceptable end-to-end experience that displays a conclusive proof of concept for our idea, of which we are proud. We feel satisfied with what the two of us achieved with this project.

What we learned

Over the course we learned about iterating our design after discussion with mentors and judges. This gave us the knowledge needed to understand the marketability of this product. Secondly, we also got to work on cool LLM tools which gave us an opening to explore the possibilities.

What's next for Senior Savvy Solutions

Future work:

Going forward, we would like to let users simply upload any product instruction manual, and watch as our cutting-edge AI technology instantly comprehends its content. Then, harnessing the power of your smartphone camera, our platform seamlessly identifies your surroundings and pinpoints the product you're interacting with. While our current prototype focuses on OCR integration for text recognition, we aim to further enhance our platform by implementing real-time video processing capabilities, allowing users to receive guidance while viewing product demonstrations.

Empowerment lies at the heart of our platform. Once the user inputs their query, our intelligent model springs into action, deciphering the nuances and intricacies of their request. Not content with mere comprehension, our model goes the extra mile, proactively seeking clarification when needed to ensure crystal-clear guidance. Whether it's directing the user to rotate or manipulate an object, pinpointing the exact button to press among a myriad of options, or simply elucidating the function of a particular button, our AI-driven solution delivers unparalleled assistance tailored to each user's unique needs. Due to severe time constraints, integration of AR was not possible.

Our vision extends beyond individual applications to embrace a broader ecosystem of seamless integration and enhanced user experiences. Through the introduction of an API version of our app, coupled with reusable UI components and plugins, we aim to revolutionize the landscape of interactive guidance for physical products. Similar to the familiar walkthroughs encountered in software applications, our solution empowers companies to craft immersive experiences that gamify user interactions, elevating the overall product experience to new heights. Not only does this approach enrich user engagement, but it also delivers substantial cost savings by reducing the reliance on manual assistance and call center resources. With our innovative solution, companies can unlock a new realm of user-centric experiences while optimizing operational efficiency and resource allocation.

Built With

Share this project:

Updates

posted an update —

Going forward, we would like to let users simply upload any product instruction manual, and watch as our cutting-edge AI technology instantly comprehends its content. Then, harnessing the power of your smartphone camera, our platform seamlessly identifies your surroundings and pinpoints the product you're interacting with. While our current prototype focuses on OCR integration for text recognition, we aim to further enhance our platform by implementing real-time video processing capabilities, allowing users to receive guidance while viewing product demonstrations.

Empowerment lies at the heart of our platform. Once the user inputs their query, our intelligent model springs into action, deciphering the nuances and intricacies of their request. Not content with mere comprehension, our model goes the extra mile, proactively seeking clarification when needed to ensure crystal-clear guidance. Whether it's directing the user to rotate or manipulate an object, pinpointing the exact button to press among a myriad of options, or simply elucidating the function of a particular button, our AI-driven solution delivers unparalleled assistance tailored to each user's unique needs. Due to severe time constraints, integration of AR was not possible.

Our vision extends beyond individual applications to embrace a broader ecosystem of seamless integration and enhanced user experiences. Through the introduction of an API version of our app, coupled with reusable UI components and plugins, we aim to revolutionize the landscape of interactive guidance for physical products. Similar to the familiar walkthroughs encountered in software applications, our solution empowers companies to craft immersive experiences that gamify user interactions, elevating the overall product experience to new heights. Not only does this approach enrich user engagement, but it also delivers substantial cost savings by reducing the reliance on manual assistance and call center resources. With our innovative solution, companies can unlock a new realm of user-centric experiences while optimizing operational efficiency and resource allocation.

Log in or sign up for Devpost to join the conversation.