-
-
First iteration of EyeQuery, so young
Inspiration
I spend a lot of my free time messing around with computer vision; whether I actually make something useful or not doesn't matter to me, I just like having fun. As I sat in my chair thinking, I decided: why not go back to my bread and butter, but this time with a unique twist to make sure I win the hackathon?
What it does
EyeQuery (EyeQ) is an OpenCV computer vision model that bridges the wide gap between static computer vision and human interaction. This program allows for humans in any field—whether it be a mechanic, technician, or anyone looking at something they don't recognize. EyeQ allows you to specify where you want the computer to look, then it sends the cropped photo to the Featherless.ai model, then it responds. No keyboard required.
How we built it
I started with small steps. First, it was the drawing tool, then implementing the Claude API -> Featherless.ai API. As I kept envisioning what this could be, I got more creative: more MediaPipe gesture detection, and using pyttsx3 and speech_recognition, and TTS and STT. I wanted it to have a camera-like functionality as well! So, I added zoom and photo capture by putting up a 3.
Challenges we ran into
The Featherless.ai models kept crashing and sometimes wouldn't respond, which took the most time out of my day. I also had a lot of issues with the reliability/accuracy of the gestures; a lot of the time the gestures wouldn't work, and if they did, it would start being really twitchy and annoying.
Accomplishments that we're proud of
As my first hackathon ever, I'm extremely proud of myself for ever coming up with a product that works and that I'm happy with. I'm also extremely proud of my planning phase. I spent about 2 hours at the start fully planning everything out and possibly failures, which definitely saved me a lot of time.
What we learned
Including API calls in my project was a first for me, and I learnt how to incorporate it properly. It also took me a while to wrap my head around having the TTS and STT work properly, and for it to be able to actually see the image and decipher what's inside of it.
What's next for EyeQ
Given more time, it could've been uploaded on a Jetson Nano, so then it could work as a true assistant. All you have to do is point/circle an object you don't recognize, and it could describe it for you instantly.
Log in or sign up for Devpost to join the conversation.