Real-time Chat Translation
Chat Translator in Action
Light and Dark Mode
Bring the world closer together
Twitch has enabled me to make friends and connections all across the globe whom I otherwise would have never met and I love the platform for this ability to connect people everywhere and bring the world closer together. Yet languages still define boundaries on Twitch and it is still rare for communities to cross-pollinate outside of their language. In today's world that keeps dividing and drawing lines, Chat Translator shows one way that we can break down these barriers and empower streamers to reach a larger audience and make the world feel smaller, friendlier, and more like home.
With Chat Translator, viewers can talk to the streamer and have conversations among each other, each in their own native tongue as if all other chatters speak the same language. It automatically detects and translates messages from 21 different languages to the currently-supported 5 languages. And as any avid Twitch user knows, Twitch chat is a little different from normal text, and full of emotes, Kappas, and partial text. These are preserved through the translations as well, in order to keep true to the original meaning as much as possible.
Chat Translator started out as one of the many open-source Twitch-integrated projects I build live on stream with viewers at the Coding Cafe as Instafluff because I had viewers from all over the world that could speak Czech, Spanish, Dutch, German, and even Japanese but were limited to chatting in English. We created ChatTranslator the Chatbot and turned it into a global Twitch bot and it currently resides in 112 different channels and has translated more than 47 million characters since September 2018. I wanted to take the lessons learned from the chatbot and make it much more natural and integrated with less chat-spam for my community and others as well, and Twitch extensions were a perfect way for me to make it happen.
I am a huge AWS fan and have used many of its services for almost a decade now in my own startups and in consulting for other businesses as well, but if not careful, it can get expensive. Chat Translator is a showcase to how, with the good architecture, AWS can be extremely cost-effective for even the penniest of the penny-pinchers.
There are several significant challenges that Chat Translator overcomes that might not be as obvious at first.
- Keeping the translation API calls budget-friendly
- Scaling real-time translation of active chat to multiple languages at the same time
- Handling Twitch Emotes in the translation process
- Sockets and real-time broadcasting of the translated message to viewers
- Listening to Timeouts/Bans of users and broadcasting the deletion of corresponding messages.
- Designing to look like the familiar Twitch chat with status and sub badges, username colors, showing emotes
API Limits & Scaling
The precursor to this extension, the ChatTranslator Chatbot, was already pushing the limits of a 15 million character monthly quota when I began working on the extension for this challenge. I made several improvements to the caching system because this extension may translate to 5 languages at the same time in a single channel so that I could keep the API costs low.
The extension connects to the translation server via web sockets, but automatically connecting every viewer to the server will overload it quickly, so I made adjustments to also keep the connection count low and only translate languages as necessary.
Twitch Chat Quirks
Twitch chat messages are not normal text. It includes emotes represented as reserved keywords and replaced at render-time to images and translation APIs do not understand the emote names and often comes back broken. Chat Translator specially handles the quirks of Twitch chat messages in the translation process in order to preserve the meaning of the original text.
Also, when a user is timed-out or banned from a channel, all messages from that user in the chat's history is deleted. Chat Translator is able to listen for these events and react as well.
Lastly, in the native Twitch chat, users often have badges reflecting their role in the channel or a special status, such as moderator/bit-cheer/subscriber/etc. including channel-specific badges. Chat Translator's design keeps these badges and other aspects of chat intact so that they do not lose any social value or cause confusion for the user compared to the native chat.
Chat Translator's Future
Even with the smart-caching and other systems put in place to help keep the API costs low, it is still a significant operational cost for one person and the bit monetization model doesn't necessarily fit well. After the challenge, I may look into a subscription model to be able to cost-scale the extension, but really the Chat Translator experience makes much more sense directly integrated into Twitch.
I believe that Twitch is a truly unique platform because of its social aspect and is strengthened by the connections made on it, which becomes even clearer at events like TwitchCon where people who have only met through the platform can meet in-person for the first time and still celebrate their long-time friendships. This is one of the reasons I love Twitch and believe that Twitch and all of the streamers and users on it would greatly benefit if they could expand their audiences and their social circles beyond language barriers.
If this extension were to win the challenge, among other things, I would like to spend the time with Twitch leadership to advocate for its native integration into Twitch before one of its competitors do.