Project Kasa

Inspiration

Ghanaian languages like Twi are still poorly supported by existing speech recognition systems. The problem becomes even more noticeable when people switch between English and Twi in the same sentence, which is common in everyday conversations in Ghana. Models that work well for English or Twi on their own often struggle when the speaker switches between the two.

We had already fine-tuned Whisper Small on English-Twi code-switched speech, which gave us a much better starting point for this problem. But the resulting model used more than 1.6 GB of RAM during inference. That is not something you can comfortably run on the mid-range Android phones that many people in Ghana use.

The Arm Create challenge gave us a reason to take the next step and actually figure out what it would take to run Project Kasa on a phone instead of a laptop. Instead of treating a model that runs well in a notebook as the finished product, we wanted to see how far we could shrink it while keeping it useful.

We were also inspired by an independent compression study on DONDO, a Conformer based ASR model for Twi and Ewe. That study found that simple quantization did not shrink the model nearly as much as expected. It made us question whether quantization alone would be enough for our model, and pushed us to look at changing the architecture itself, not just the number format.

What it does

Project Kasa is an English-Twi code-switched speech recognition system built for Ghanaian speech. It is based on a fine-tuned Whisper model adapted to recognise speech where people naturally move between English and Twi mid sentence.

For this challenge, we focused on making that model small enough and light enough to actually deploy on a phone.

Our starting point was the fine-tuned Whisper Small model. We measured its word error rate, character error rate, inference speed, and peak memory use, then used those numbers as the baseline for everything else.

From there we tried different ways to shrink the model, including cutting down the number of transformer layers and applying quantization. The goal was never just to make the model as small as possible. It was to find a good balance between size, memory, speed, and how well it still understood the speech.

How we built it

We started by fine-tuning Whisper Small on our English-Twi code-switched dataset. Once training was done, we built a baseline using a held out test set of more than 1,600 samples.

Then we tried three main approaches.

Architecture depth reduction. We built smaller versions of the model by keeping an evenly spaced subset of the transformer layers, rather than training a new small model from scratch. We warm started each smaller model from the trained Whisper Small and then fine-tuned it again on our data.

ONNX Runtime quantization. We exported the model to ONNX and quantized it using settings aimed at Arm64 devices specifically, since the phones we actually want this to run on use Arm chips, not the x86 CPUs most development happens on.

whisper.cpp and GGML quantization. We also converted the model into GGML format and tested lower bit quantized versions through whisper.cpp, which is closer to what real on-device Whisper apps use in practice.

Every version was tested with the same metrics and the same test set, so we could compare them fairly instead of judging size or accuracy on their own.

Challenges we ran into

A lot of the work turned out to be getting a model out of a research notebook and into something that could actually be deployed. A few things we ran into along the way:

Model export. The ONNX export tool could not automatically tell that our locally saved checkpoint was a transformers model. We had to explicitly point it in the right direction using the command line export path instead.

Quantized file names. ONNX Runtime saves quantized models as new files with "_quantized" added to the name instead of replacing the originals. We had to update our loading code to point at the right files.

Generation settings. An outdated generation_config file broke transcription entirely. Our first fix, copying the settings from the original Whisper model, got rid of the error but also quietly changed decoding behaviour and made the model perform worse. We eventually found that fixing only the one broken setting, instead of replacing the whole file, was the right move.

whisper.cpp on Windows. Getting whisper.cpp running took extra setup work. We had to install CMake and the Microsoft build tools, figure out the right arguments for the conversion script, and sort out path and working directory issues when calling the compiled program from a notebook.

None of this was conceptually difficult, but it was a good reminder that shipping a model is a very different job from training one. The model, the export format, the runtime, and the hardware all have to actually agree with each other.

Accomplishments that we're proud of

The result we are most proud of is our best compressed model. After cutting the model down to six encoder and six decoder layers and quantizing it with whisper.cpp, the model shrank to 151 MB on disk and used 429.9 MB of peak memory during inference, down from 1,607 MB for the original. That is a quarter of the memory the full model needed. Word error rate only went up from 11.6 percent to 13.2 percent, a small trade for that much smaller footprint.

We also fine-tuned a Whisper Tiny model and quantized it with whisper.cpp. That version came out to just 41 MB and 237.5 MB of peak memory, and it actually scored a lower word error rate than our full sized baseline, 8.1 percent compared to 11.6 percent. A model roughly a sixth the size ended up more accurate, which was not something we expected going in.

We are also proud that we tested three genuinely different approaches instead of assuming one technique would be enough. Comparing depth reduction, ONNX Runtime quantization, and whisper.cpp quantization side by side gave us a much clearer picture of what actually works and what does not.

Most importantly, this project moved us from just training a model to actually solving a deployment problem. We now have a real, measured picture of what it takes to get an English-Twi ASR model out of a research checkpoint and onto a phone.

What we learned

The biggest lesson was that quantization is not always the right place to start when shrinking a speech model. Based on what we saw in related work on similar architectures, straightforward quantization mostly targets the linear layers in a model, while Whisper does a lot of its work in convolution and attention layers instead. That is part of why it did not always shrink things as much as we expected.

We also learned that speed was never really our problem. Our baseline model was already running at around five times real time on CPU before we optimized anything. That told us memory and model size were the real bottlenecks for getting this onto a phone, not raw speed.

That changed how we approached the whole project. Instead of chasing faster inference, we focused first on cutting the number of parameters through depth reduction, and treated quantization as an extra layer of compression on top of that.

Another lesson was that you have to think about the target device from the start. Settings tuned for an x86 CPU are not automatically right for an Arm chip, and a model that runs fine in a Python notebook can still need a lot of extra work before it runs on a phone.

What's next for Project Kasa

The next step is pushing further toward real mobile deployment. Our layer reduction experiments have shown there is real room to shrink the model, and we want to keep testing where that reduction starts costing too much accuracy.

We noticed some real differences in accuracy between the ONNX Runtime and whisper.cpp versions of the same quantized model, and we want to dig into exactly why before drawing firm conclusions from it.

We also want to move past desktop benchmarking and test these models on real Arm based phones. That is the only way to properly measure RAM use, latency, battery impact, and real time performance under actual conditions.

Longer term, we want Project Kasa to be more than a compressed model. The goal is real speech technology for Ghanaian languages that runs directly on the phones people already own, without needing a powerful computer or a constant internet connection.