Inspiration
In 2022, while working as a Research Assistant at the Institute for Intelligent Systems, I led a team that developed a Python script to extract Twitter sentiment towards foreigners in South Africa, using keywords like "foreigners" and "Put South Africa First." This research highlighted the prevalence of xenophobic sentiments in the country. Inspired by the project and realising that the problem still persists, I decided to create a platform to raise awareness using 3D visualisation and audio. My inspiration came from a similar project called "Voices of Racism" by the Human Rights Commission of New Zealand. Unfortunately, the website was taken down, but I was able to learn how to create something similar through available tutorials.
What it does
The Xenophobic Voices website is designed to give users a chilling visual and audio representation of xenophobic sentiments. When a user clicks on the grid of lines, a 3D head model breaks through from the darkness, symbolising how these hateful views are hidden but always present. As the head appears, it begins to speak negative sentiments about foreigners, with dark, unsettling background audio that intensifies the ominous feeling. The animation and audio work together to show how frightening xenophobia can be, making the issue more palpable for the user. This immersive experience is meant to foster empathy and awareness by vividly portraying the impact of xenophobia.
How we built it
I started by extracting sentiment data from Twitter and identifying keywords related to xenophobia. Then, I researched web technologies such as Three.js and WebGPU to create 3D visualisations. I spent several days building the initial 3D grid using shaders, which responds to the depth of the scene. I also explored how to animate a 3D head model, which included overcoming challenges related to importing and converting formats. After several failed attempts with FaceCap, I found a suitable 3D model and used Autodesk Maya's SDK to convert the animation data into a format compatible with web browsers. Audio synchronisation was achieved through Rhubarb Lip Sync and AWS Polly, which generated the speech to animate the 3D model’s mouth movements.
Challenges we ran into
One of the major challenges was animating the 3D head model. Initially, I tried using FaceCap to record mouth movements, but the format it generated was incompatible with web browsers. I also encountered difficulties in synchronising audio with the model’s mouth movements. The process required extensive research, multiple trials, and revisiting Python to create a script that would allow for conversion into a suitable format. Additionally, syncing audio to the model’s mouth movements required a detailed understanding of lip-sync algorithms, which took considerable time to master.
Accomplishments that we're proud of
We are proud of successfully animating the 3D head model and syncing the audio with mouth movements, which was a significant technical challenge. The platform’s ability to visually represent and narrate the negative sentiments towards foreigners through a 3D model and audio is a key accomplishment. Additionally, deploying the website on AWS EC2 and ensuring that it was accessible to a global audience was a critical achievement.
What we learned
Throughout the development process, we learned how to work with complex 3D graphics technologies like Three.js and WebGPU, and we gained a deeper understanding of lip-syncing algorithms and their application in animation. The project also taught us how to integrate multiple web technologies to create an interactive experience. On the deployment side, we learned how to use AWS EC2 and Nginx to host a live website and manage its deployment effectively. Most importantly, we learned how to overcome challenges through persistence, research, and collaboration with tools like AI.
We also learned how important it is to iterate on user experience (UX) and visual appeal, especially when it comes to animations. Making the head’s mouth movements more crisp and realistic will require more fine-tuning, which has given us deeper insights into animation and graphics work.
In addition, we gained a valuable understanding of sentiment analysis and natural language processing (NLP), particularly in the context of social issues. Analysing sentiments from social media posts provided us with a more nuanced view of how xenophobia persists in South African society today. The project was not only a technical challenge but also a reminder of the ongoing social issue of xenophobia that continues to affect many people. Through this process, we realised the power of technology in raising awareness and prompting conversations about these critical societal issues.
We also learned a great deal about collaborative tools like Git and GitHub, which were essential in managing the codebase and facilitating version control throughout the project. These tools allowed us to work efficiently and ensure that everyone was on the same page, making it easier to track changes and collaborate on different aspects of the project.
Additionally, we gained experience with web browser testing, which helped ensure that the app ran smoothly across different browsers and devices. This aspect of testing was crucial in delivering a seamless user experience, as we had to ensure compatibility with various browsers and fix bugs related to performance and visuals.
What's next for Xenophobic Voices
Looking ahead, we aim to enhance the Xenophobic Voices project by expanding the range of voices and languages, making it more representative of diverse perspectives on xenophobia. We also plan to implement more interactive features, such as allowing users to submit their own experiences or thoughts on the issue. Additionally, we aim to refine the animation for a more human-like experience, particularly by making the mouth movements more natural and crisp. We will also integrate educational modals to explain why some of the displayed sentiments are considered negative or xenophobic, helping users understand the context behind the message. The ultimate goal is to continue using this platform as a tool for social change, raising awareness, and fostering understanding in South Africa and beyond.
Log in or sign up for Devpost to join the conversation.