Grapheme: An AI-powered Translation/Transcription Tool

Inspiration

Language barriers hinder effective communication, emphasizing the need for accurate translation tools that facilitate understanding across diverse languages and cultures. Current solutions, i.e., Google Translate, add an extra layer of latency via networking. Another option includes: Offline options for Google Translate exist (i.e., downloading the language packs for offline translation) That would require needing to download something ahead of time before doing any real-time translations.

What it does

Our project allows for dynamic language detection, transcription and/or translation using nothing but local system hardware (i.e., CPU & GPU), eliminating as much overhead as possible.

How we built it

We used Visual Studio, CMake, and Premake to set up our project. Our application uses a 2-layered system, 1 for the application, and 1 for a "core" library, which handles communication between our 3 dependencies. Namely, whisper.cpp, json & miniaudio. The API we made includes wrappers written in both C & C++ to allow for a portable, dynamically linked library to be called from via the binary itself.

Challenges we ran into

For videos or audio containing rapid switches between 2 or more different languages, translations became noticeably inaccurate.
Our libraries occasionally lacked support for specific languages and dialects (i.e., North Korean, Akkadian or other obscure languages), and can only translate to English only, leading to challenges in maintaining translation accuracy across diverse linguistic pairs.
Optimizing translation speed and responsiveness became crucial, as larger datasets resulted in noticeable delays during real-time translation requests. GPU-based translations/transcriptions were noticeably more performant CPU performance is noticeably slower, but we can address that later through micro optimizations