Sparky - The Hardware Guru

Inspiration

Sparky was inspired by our friend, a true hardware enthusiast with extensive knowledge of tools and electronics. His ability to troubleshoot and guide us through hardware challenges sparked the idea for an interactive assistant that would simplify the process of identifying and understanding hardware components. We wanted to create a resource that could help users feel confident in their hardware knowledge, no matter their skill level.

What it does

Sparky - The Hardware Guru is an interactive assistant designed to help users identify and understand various hardware and electronic components. By leveraging voice recognition and image processing, Sparky can respond to questions about tools and electronics, providing concise, layman-friendly explanations. Users can either ask about specific tools or show an image of a component, and Sparky will identify it and offer relevant information to enhance their knowledge.

How we built it

We built Sparky using a Raspberry Pi as the central hub, integrating a microphone for voice input, a speaker for audio output, and a camera for image recognition. The system employs software algorithms for processing voice commands and analyzing images of hardware items. The components are housed in a custom 3D-printed model, giving Sparky a unique and engaging design. We utilized libraries for speech recognition and computer vision to facilitate the interaction.

Challenges we ran into

While developing Sparky, we encountered several challenges, especially in gathering the appropriate dataset for a custom training model using YOLO. Achieving satisfactory accuracy required multiple training sessions of the model. Another challenge was ensuring the system was interactive and communicated effectively within a domain-specific context. Furthermore, integrating various events and deploying the system on the hardware added complexities, particularly given our limited timeframe.

Accomplishments that we're proud of

We are proud to have fully assembled the assistant to effectively respond to user inquiries and provide live detection of tools, along with accurate summaries. Achieving an accuracy rate mAP50: 82% and mAP50-95: 67% across six different classes is a significant milestone for us. This accomplishment showcases our ability to blend technology and user experience, allowing Sparky to serve as a reliable assistant in the realm of hardware and electronics.

What we learned 

We discovered creative methods to generate content specific to our domain using generative AI and learned how to activate computer vision models directly within the chatbot. also through this project we got a better how a multi-model system works.

What's next for Sparky - The Hardware Guru

Moving forward, we plan to enhance Sparky's capabilities by improving accuracy through the addition of more images and expanding the dataset. We aim to enable Sparky to recognize unknown tools by providing images for on-the-spot training. Currently, Sparky can identify certain tools but lacks knowledge of their appearances. By supplying images, we can enhance its understanding and facilitate real-time learning. Additionally, we need to refine the user interface on the website and integrate the computer vision model. We also envision adding motion capabilities by incorporating motors or servos into the hardware, which would enhance its aesthetic and functionality.

In conclusion, Sparky is more than just a project; it’s a transformative tool designed to empower users and demystify the world of hardware and electronics. By delivering clear, accessible information and unwavering support, we aim to inspire confidence and curiosity in anyone facing hardware-related challenges. With Sparky by your side, you’ll feel equipped to tackle any task, turning obstacles into opportunities for learning and growth.