Inspiration
Have you read Harry Potter? At Hogwarts, the main gate of each House has a portrait that watches it, and when you want to get in, you have to interact with that portrait to get in. For the Ravenclaw house, you would have to answer a riddle to get past the door. So, here comes my product.
What it does
When a person is actively looking at the camera, the system will ask a riddle for that person. By analyzing the person's face and voice, the system can determine if that user can go through or not based on a list of approved people to pass.
How we built it
The door lock system consists of a Rubik Pi 3 running both the EyeGaze and the MediaPipe-Face-Detection for face detection and face gaze detection. I also used the Gemini API to generate Medieval English-style questions, and use ElevenLabs API to get the best voice for the theme. I also created a web app hosted at my own home server to enroll faces if the owner of the door decided to add new users.
Challenges we ran into
At first, I thought that I could run the voice classification, gaze detection and face detection on-device, but apparently, there is no model for voice classification for the Rubik Pi 3. Also, since there is no MLH Hardware Lab, and I didn't get enough hardware beforehand, I lacked parts that I need to complete the product. Finally, the product needs a screen and a speaker to at least output the face-box that we normally see on Computer Vision tutorial.
Accomplishments that we're proud of
I came here just to try out the capability of the board, and seeing a small edge devices with such powerful computational power enlightens me.
What we learned
I learned how to deploy models on the NPU of the specialized Qualcomm chipset, and I learn how to deploy web app on server.
What's next for SmartLock
I would like to complete the product more, and maybe add a monitor to fully demonstrate how the Rubik Pi 3 detects gaze and face. I also wants to implement the voice classification in to further tightening the security for people who wants to get past the door.
Log in or sign up for Devpost to join the conversation.