-
-
Azure OCR from hand written note
-
chat
-
hand written note
-
Azure Afrikaans TTS
Empangeni-LAN-Chat: Bridging Communication Gaps in South Africa with AI
Inspiration
In Empangeni, South Africa, language diversity is a beautiful asset but also a communication challenge. Misunderstandings due to communication barriers are common, hindering everything from casual conversations to business negotiations. We saw a need for a tool that could empower individuals to connect and understand each other better, regardless of their native tongue nor background. That's why we created Empangeni-LAN-Chat.
What it does
Empangeni-LAN-Chat is an AI-powered LAN chat application designed for inclusivity. It provides:
Real-time translation: Seamlessly translates between Afrikaans, English, and Zulu, the three most common languages in our community. Voice support: Offers text-to-speech capabilities, making the app accessible to those with visual impairments or literacy challenges. OCR for handwritten notes: Digitizes handwritten notes, enabling their inclusion in the chat and translation process. Rephrasing and tone adjustment: Helps users craft messages that are clear, respectful, and culturally sensitive. Conversation analysis: Provides insights into communication patterns, helping users understand and improve their interactions. How we built it We leveraged the power of cutting-edge AI technologies:
Gemini: For advanced language understanding, culturally nuanced translation, and rephrasing capabilities. Azure Cognitive Services: For accurate speech-to-text, text-to-speech, and OCR functionalities. Flask: To build a robust and scalable backend. HTML, CSS, and JavaScript: To create a user-friendly and accessible frontend.
Challenges we ran into
Limited Afrikaans/Zulu proficiency in Azure LLMs: While Azure offers powerful language models, we found their proficiency in Afrikaans and Zulu to be less robust than needed for our Empangeni community. This limitation impacted the accuracy and cultural sensitivity of translations. Langchain's lack of Gemini support: Integrating Gemini with other tools proved challenging due to Langchain's lack of direct support for this model. Building custom pipelines required significant effort and expertise. Lack of cost-effective TPU acceleration on Azure: To fine-tune our models for optimal performance in Afrikaans and Zulu, we require TPU acceleration. While Azure offers GPUs, we found that Vertex AI's TPUs provided a more cost-effective and time-efficient solution for our specific needs. This highlights an area where Azure could potentially improve its offerings to better support projects like ours. GitHub Copilot's open-source friction: While Copilot can be a helpful tool, we encountered instances where it introduced errors into our code, creating unexpected debugging challenges. Azure Model Zoo(or lack there of) We found the model zoo on Azure to be significantly less comprehensive compared to Vertex AI. The lack of readily available community projects and implementations made it challenging to find relevant starting points for our specific use case in Empangeni. This resulted in increased development time and effort.
Accomplishments that we're proud of
Effective communication bridge: We successfully built a tool that facilitates clear communication across language, emotional, cultural, educational and frame of reference barriers. Culturally sensitive design: By incorporating Gemini's understanding of cultural nuances, we ensured respectful and accurate translations. User-friendly interface: The application is intuitive and accessible to individuals with diverse technical backgrounds.
What we learned
The power of AI for social good: We witnessed firsthand how AI can be used to address real-world challenges and create positive social impact. The importance of user-centric design: Building an application that is truly useful requires a deep understanding of the target users and their needs. The value of collaboration: Working together as a team allowed us to leverage each other's strengths and overcome challenges effectively.
What's next for Empangeni LAN-chat
Expand language support: Add support for more languages spoken in South Africa. Enhance accessibility: Integrate features like text-to-Braille and sign language support. Develop a mobile app: Make the application accessible on smartphones and tablets. Explore offline functionality: Enable basic functionality even without an internet connection. We believe Empangeni-LAN-Chat has the potential to transform communication within our community and beyond. We're excited to continue developing this project and making a positive impact on the lives of others.
GitHub Copilot Integration and Project Philosophy
While the Azure Hackathon guidelines suggested the use of GitHub Copilot, my experience integrating it into the backend Python development of Empangeni-LAN-Chat proved challenging. As documented in the attached JSON conversation file, Copilot frequently introduced errors and inefficiencies, hindering rather than accelerating progress. This resulted in significant debugging and manual correction, demonstrating its limitations in this particular development context. However, the core purpose of Empangeni-LAN-Chat transcends the tools used in its creation. This project was born from a deep-seated desire to serve my community in Empangeni, KwaZulu-Natal, where language barriers and access to information can be significant hurdles. It was built with practicality and accessibility in mind, leveraging the most effective AI models from Vertex AI and Azure Cognitive Services to create a tool that directly addresses these challenges. This application is not about aesthetics or impressing judges. It's about bridging communication gaps and empowering individuals within my community. By integrating real-time translation, voice support, and OCR capabilities, it provides a vital communication bridge for schools, local clients, and anyone seeking to connect with others, regardless of language proficiency or literacy levels. Empangeni-LAN-Chat is open-source and packaged for easy download and use. If others find value in this tool, I encourage them to utilize and contribute to its development. Its primary focus remains serving the people of Empangeni, but its potential to benefit a wider audience is undeniable. This project is a testament to the power of AI to create tangible solutions for real-world problems. While the integration of GitHub Copilot did not meet my expectations, the project's core purpose and functionality remain strong, driven by a genuine desire to make a difference.
Log in or sign up for Devpost to join the conversation.