Inspiration
Less than two weeks ago, American rapper PnB Rock was shot and murdered after his location was revealed by his girlfriend's Instagram story. This is not the first time this has happened: fellow rapper Pop Smoke was murdered in 2020 in a home invasion after the address of his rental house in LA was revealed on Instagram. The sharing of location-sensitive information on social media poses a risk to the lives of celebrities and the rest of us on the internet. We created wya to help solve this problem.
What it does
wya is a Chrome extension that alerts posters if their photo contains location-sensitive information. It's primarily intended for Instagram, but it can be used with any social media app. Before posting, the user simply uploads their photo to wya. Then, wya scans the image for location-sensitive information—for example, an easily recognizable landmark, or text giving an address—and makes a recommendation as to whether the user should post that photo.
How we built it
The frontend of the Chrome extension is built using HTML, JavaScript, CSS and Bootstrap. The backend is done using Java and Spring Boot. Specifically, the Java backend calls a Python script that does the image processing. This Python script leverages computer vision (landmark and OCR) from the Google Cloud Vision API. The NLP post-processing of the text obtained by OCR is processed using the Named Entity Recognition models within the Spacy-NLP library.
Challenges we ran into
We ran into challenges both with the Python script and with the relationship between the frontend and backend. On the Python side, a challenge we ran into was detecting addresses within the extracted text from an image; in particular, there does not seem to be any library or API that specifically does this, which is difficult given how varied in form addresses can be. We were able to solve this using Named Entity Recognition and recognizing that certain combinations of words (ie. a country + a building name) would likely be an address. On the frontend/backend side, a challenge was connecting the two—specifically, having to deal with the restrictions of building a Chrome extension. Other challenges included executing a Python script from the Java backend and passing the output as a JSON to the frontend, and sending image files between the frontend and backend and dynamically updating the frontend.
Accomplishments that we're proud of
Putting together a multi-layered extension/web app that performs what we had envisioned. Learning and applying numerous skills in frontend, backend, scripting, and working with APIs.
What we learned
We learned how to create a Chrome extension, set up and connect it with a Java/Spring Boot backend, and write and execute a Python script calling a variety of APIs. In this process, we learned a variety of new tools, from Google Cloud API to Java Springboot to Python Spacy-NLP. Importantly, we also learned skills in teamwork and collaboration: setting our project scope, prioritizing our tasks, and working to bring everything together.
What's next for wya
Our next steps for wya are (1) integration, (2) improved address recognition, and (3) delayed posting. First, we plan to integrate this extension with Instagram, particularly for mobile use. Our long term vision is that this extension works seamlessly in the background and simply pops up whenever a user tries to post a photo or story with location-sensitive information. Second, plan to improve address recognition using a custom trained model. Third, we plan to add an additional feature so that if the user wants to post a photo with potentially revealing information, they can still do so, but our app will automatically delay the posting until their location changes.
Log in or sign up for Devpost to join the conversation.