Here is the revised submission, formatted to match your style.
Inspiration Facial classifiers are future tech that seems far away and yet most people don't realise just how common and easy it is to pull a person's face from local to internet. We provide this example as to raise awareness of internet privacy, and why it's important for us to be up to date with technology for better or worse. Tech is advancing at a pace where there's too much for us to keep up knowing about and yet, our policies don't reflect as such. Hence, we wanted to pose an example of what happens when given our sample population of hackathon people, a linkedin profile which can view most faces, and a powerful modern facial classifier can do with tech stacks that were made approximately 2 years ago.
What it does Fetches faces from a live video feed, searches them against linkedin faces that we found from the devpost hackathon list and sees if we can match a face to a linkedin profile. It also tracks unknown people and builds a profile for them in real time as they move through the venue.
How we built it ZED 2 Stereo Camera for depth sensing and 3D positional tracking to ensure we only scan real humans and not photos on screens.
InsightFace (ArcFace) running on ONNX Runtime GPU to extract 512 dimensional vectors from faces in milliseconds.
GFPGAN for super resolution to upscale blurry faces at a distance so the recognizer can actually see them.
MongoDB to store the vector embeddings and thumbnail snapshots for our recognition engine.
Streamlit for the dashboard where we can view live data, merge duplicate identities, and manage the database.
Flask to stream the video feed from our backend detection script to the frontend dashboard.
Challenges we ran into Model selection was tough because we had to find one that balanced speed with accuracy on a consumer GPU.
How depth calculated was tricky since the camera loses precision at distance so we had to switch to neural object detection.
Performance issues at first with the frame rate dropping when we tried to process every single frame instead of batching.
Merging faces was a logic nightmare because the database would create a new person for every slight angle change.
Model too eager to generate a new face so we had to implement a temporal voting system to smooth out the predictions.
Set limits on guesses model can make by adding strict blur checks and pose estimation guards to stop bad data.
Accomplishments that we're proud of Generating a face from a blurry crop using generative AI in real time without killing our frame rate. We are also proud of the merge workflow where we can teach the model who someone is just by grouping their photos in the dashboard.
What we learned We learned that the gap between open source code and a surveillance state is terrifyingly small. The tools we used are free and accessible to anyone with a decent computer. We also learned a lot about vector search and how your face is just a string of numbers to a computer.
What's next for FNR Facial-Net Recognizer Improving the model to handle profile views better without needing manual merges. We also want to find ways to combat the facial recognizer so people can protect their own privacy, maybe by researching adversarial patches that confuse the AI.
Log in or sign up for Devpost to join the conversation.