Eclipse

Inspiration

In the past few years, there have been numerous developments with computer vision, However we would like to draw specific to attention to models such as Clearview.ai which have been used by governments and billionaires alike around the globe to track everyone. They accomplish this by comparing photos of targets against their near endless online database of over twenty billion photos. Even if you don't consider yourself a consumer of social media or someone with a large (or any at all) online presence, they can still track you by finding you in the background of photos. Because of their vast reach, the New York Times said that Clearview "Might End Privacy as We Know It." Giving anyone, even the government, this much information about us is perilous and a slippery slope to more authoritarian activity. We know that power corrupts and "Absolute power corrupts absolutely," as Lord Acton has said, and having a tool like Clearview.ai is as close to absolute power one might have in this digital age.

But the US is different right? We have a balance of powers, elected officials, and a judicial system. However, we can't just rest on our laurels. There have been decisions of late in the US that have challenged our rights and freedoms as American citizens. Lawful citizens have been deported without due process, a direct violation of the fourth amendment. It seems like there are no guarantees that our government could ethically use these tools nor that they could regulate companies using and creating these tools.

In addition, even if we could trust applications like Clearview in certain hands, there is no guarantee that it stays in those hands we trust. In February 2020, Clearview had a data breach which exposed a private list of their customers. Two months later after they had claimed they had fixed their security flaw, Mossab Hussein found secret keys and credentials in their exposed source code. All of this proves that ALL of our privacy is at risk and that the tools that could be used for good could easily fall into the wrong hands.

We wanted to help mediate these invasions of privacy and the impact of data breaches by creating Eclipse. Eclipse blurs private details, such as peoples' faces in the background, potential credit card details, and street names that would help identify the other people and their location. Eclipse isn't just about protecting your own privacy, but also the privacy and rights of others.

How we built it

To create Eclipse, we crafted three models to accomplish our goals. For first model, we fine-tuned YOLOv11 on human pose data, in order to identify humans in images and detect their eyes to blur to maintain anonymity. The second model was a fine-tuned FaceNet model (which was a combination of P-, R-, and O-nets to extract faces and then passed through a ResNet) to perform facial recognition, so that the subject's face would not be blurred by the first model. The last model was another YOLOv11 model that was fine-tuned on object detection of license plates, street signs, credit cards, and IDs. We built the app using HTML, CSS, and Typescript, using the Tauri framework and captured video feed via JavaScript APIs. Then called code in our Rust backend to communicate through a TCP Web Socket with our Java servers in order to run the YOLO models and then passed it through another Web Socket to Python servers to run the facial recognition models. We used Rust in the backend and tried to optimize inference in our models to make the program as light as possible.

Challenges we ran into

There were a lot of languages and servers... We used over 7 languages, and it was very hard to connect them. We had to send settings, images, and even videos through these different languages, using frameworks like opencv and webhooks. It was also difficult to just manage all of these models with limited compute power and limited time. Gathering data was also a challenge specifically for the street signs and credit cards which we had to gather for the YOLOv11 model.

Accomplishments and takeaways

We are proud of getting all of this actually functioning. We knew it was an ambitious idea to start out with, and we weren't sure if it was actually going to come together by the end, but it was able to. We learned about how to implement many different languages together to create a cohesive product, and we learned a lot about using videos in applications, considering how much of our project was based around that.

What's next for Eclipse

We could optimize it more, and make it an actual application that people could download. We could also make it something that would work on phones, so users could just download it and use it for photos or videos for social media with as little inconvenience as possible.