Inspiration
Transcription and diarization are not new concepts, but just like many other features, companies heavily gatekeep or limit access to tools that should be free. Phones are underestimated for how much they can perform, and running ML models is not fiction anymore, and this is where it started with me, experimenting with using a phone's capabilities. Even mid-range phones can now run billion-parameter models without consuming many resources, more or less like playing a video game. I was personally shocked at the results with my project, high accuracy and on par with cloud services if not better sometimes. The emulator, which is used for the demo, was on a 2gb of ram constraint yet it was able to run medium whisper model!
What it does
The app transcribes, diarize, and translates fully locally and offline. It gives model choice freedom to the user, lighter or heavier, according to their needs. It also provides basic editing tools for video and media (more coming in the future, it's just an mvp now), with the ability to export with burned-in captions or get the files in various text formats, .srt, .vtt, .tml, .fcpxml and .xml for external work.
How I built it
The development took a rollercoaster of decisions. I initially used Flutter to target both iOS and Android, then favored Android due to time constraints:
- Transcription: Whisper models running on the whisper_ggml_plus runtime engine wrapper.
- Diarization: A lean local mix of pyannote 3.0 and CAM++, utilizing the ONNX runtime engine.
- Translation: Integrated Google ML Kit for ease of use and good performance.
- Audio & Video Processing: Native MediaCodec for audio extraction (avoiding FFmpeg licensing issues) and Media3 Transformer for video rendering/exporting.
Challenges I ran into
The road was not clear. At many times I was asking myself: Am I running into a physical limitation of existing tools, or am I doing something wrong? The code compiled and gave confident results, but they weren't quite right. Despite limited time, I had to brute-force my solutions:
- Tested industry-standard tools like GTCRN, but results were unacceptable.
- Tuned dials up and down for pyannote and Whisper, introduced CAM++, and analyzed bit-identical files down to milliseconds of silence between speakers. Discovered the audio decoder was reading incorrect sample rates from files which led me to fork the native whisper engine directly because it was blind to my settings all along.
Accomplishments that I'm proud of
Finally, I achieved a good enough result. regardless, they're probabilistic models and they can never be 100% right, because of this, I tried to make editing extremly easy and intutivie to undestand and deal with. With unlimited exports, no capped duration, and no monthly recurrent fees.
What I learned
A lot. I never dealt with these specific tools before, putting it under time pressure and starting three weeks late to the hackathon made me learn not only to ship even more quickly, but to also accelerate my learning methods. That regardless of the pressure and decision fatigue I faced all throughout, it was definitely worth it seeing the project come to life. I learned to also use AI along the way. If I don't , it's a losing race. The app would still never come to life with one prompt, not even with a hundred if the person doesn't know what they're doing.
What's next for Argand
Argand has some top-level secret ideas. It's only the tip of the iceberg for me, possibilities are still being investigated. As for the obvious, I believe I could still take it up a notch all around.
- Expand video editing capabilities.
- Support additional export formats.
- Publish cross-platform on both iOS and Android.
- Offer more local AI model options for users.
Log in or sign up for Devpost to join the conversation.