Inspiration
Women encounter extensive harassment and bullying across chat applications, including platforms like WhatsApp, Telegram, and Snapchat, and even professional tools like Zoom and MS Teams and even when providing customer services through simple chat applications. Many report receiving unsolicited sexual messages, explicit images, and unwelcome advances. Frequently, women face pressure to send or receive personal photos or inappropriate messages early in conversations, which often drives them away from these platforms. Young women are sometimes made to feel that online sexual harassment is a "normal" part of growing up or merely a joke, even though options exist for reporting and blocking such behavior. Alarmingly, few chat applications have mechanisms to prevent or flag inappropriate content at the outset—such as flagging inappropriate messages as they’re sent. And in the current hackathon project, this is the issue that is being addressed.
What it does
To overcome the limitations of existing chat applications that have been discussed in the previous section, I have built an application called Clean Talk as part of this hackathon. The objective of this application is to create a harassment-free chat experience by automatically flagging potentially inappropriate messages and images before they are delivered to the recipient. This proactive approach ensures that users, particularly women are protected from offensive or unwanted content, making it useful in school environments, customer services, and professional chat applications where a respectful communication standard is essential.
How we built it
To build the Clean Talk, a Python-based Socket.IO chat application that is capable of sending and receiving both text messages and images is first built. Additionally, on the server side, we integrated Gemini AI, this Gemini model is then capable of identifying inappropriate or offensive content and flags any problematic text before it reaches the recipient, ensuring a safer and more respectful chat environment.
Challenges we ran into
One of the key challenges that I faced was to select the right Gemini AI model for effective content moderation. Initially, the Gemini-1.5-Flash model was used to identify inappropriate content, but it struggled to consistently identify harassing content. This model required extensive fine-tuning with a dataset of potential harassment messages to improve its accuracy. After experimenting with several models, it was found that Gemini-1.5-Flash-8B was significantly more effective, as it could identify explicit content with greater precision, reducing the need for additional training. This adjustment improved the ability to proactively flag inappropriate messages, enhancing the chat application's reliability and effectiveness.
Accomplishments that we're proud of
There are several accomplishments, some of them are worth highlighting and explained below: 1) Building a Safe, User-friendly Chat Application:- With this hackathon, a harassment-free chat application that proactively identifies and flags inappropriate content before it reaches users was built. This project creates a safer digital space, especially for schools, professional environments, and customer services, and tries to prevent women from facing harassing messages.
2) Integration of AI:- This hackathon requires us to make use of Gemini AI which has been accomplished in this project since I was able to effectively integrate Gemini AI to analyze and flag harmful content, improving the reliability of digital communication.
3) Overcoming Challenges:- Through trial and error with different AI models, the best-performing version for content analysis was identified, significantly enhancing the application’s functionality and making it a highly effective tool.
4) User-Centered Design: This application design prioritizes the safety and well-being of women, and we’re proud of creating a platform that promotes respect and professionalism in online interactions.
What we learned
1) Importance of Model Selection and Training: Choosing the right AI model is crucial for the accurate identification of harassing content.
2) Balancing Sensitivity and Accuracy: Implementing content moderation requires balancing the system’s sensitivity to flag offensive content without generating false positives, which requires thoughtful model tuning and testing.
3) User Privacy and AI Ethics: Working on content moderation raised important discussions about privacy and ethical AI use, which further helped to approach the design with sensitivity to both user safety and privacy concerns.
What's next for Clean Talk
The initial aim or objective of this project was to build a simple chat application that protects users, especially women in customer service roles, from facing online bullying and harassment by angry customers. However, this application can also be extended for: 1) School and Professional Environments:- We have several applications such as Zoom, MS Teams and so on that are widely used in professional environments for meetings, chatting, and so on. A more advanced version of the Clean Talk application can also be used in professional environments thus preventing female employees or school children from facing any kind of online harassment.
2) Multimedia Support:- Right now the application works by identifying inappropriate or harassing text messages. This can however be extended to handle images, videos, and documents.
3) User-Driven Customization:- We want to allow businesses and individual users to customize the sensitivity and criteria for flagged content. This would give organizations more control over maintaining a respectful communication environment tailored to their specific needs.
Log in or sign up for Devpost to join the conversation.