Inspiration
Cloud writing assistants are everywhere, but they assume reliable, affordable internet. Across much of Africa, many people write on low-cost laptops with limited connectivity. Even with a connection, sending your writing to a remote server is a privacy tradeoff.
I wanted a writing assistant that works entirely on-device no internet, no subscription, and no data leaving the machine.
What it does
This is an offline writing assistant with two features in a Grammarly-style editor
Proofread: As you type, it checks grammar in the background and marks issues with inline underlines. A sidebar shows the original text, the suggested correction, and accept/dismiss controls. Paraphrase: Highlight a sentence and press Paraphrase to generate three alternative rewrites while preserving the meaning.
Everything runs locally. There are no network calls during inference.
How we built it
Frontend: Vite + React with a contentEditable editor, formatting toolbar, inline suggestions, an accept/dismiss sidebar, and a score indicator. Grammar checks are debounced and only rerun for sentences that changed.
Backend: FastAPI. Grammar correction uses CoEDIT, a FLAN-T5 editing model. I split text into sentences with wtpsplit, run each through CoEDIT, then generate a word-level diff and map corrections to character offsets for the editor.
Paraphrasing: Llama-3.2-1B-Instruct, converted to GGUF and quantized to Q4_K_M, runs through llama.cpp. A defensive parser converts imperfect model output into clean rewrite options.
Model tooling: llama.cpp's converter, llama-quantize, and llama-bench.
Challenges
I initially wanted CoEDIT as a GGUF model, but T5 generation doesn't work reliably in modern llama.cpp. The architecture loads, but llama-cli produces 0 tokens and llama-server hangs. I caught this early with a spike and switched to keeping CoEDIT as a regular model, while using Llama-3.2-1B for the GGUF/llama.cpp path.
The 1B model often produced malformed structured output, so I built a defensive parser that extracts valid results without letting bad JSON reach the UI.
CPU performance was another bottleneck. Beam search made grammar checks take ~40 seconds per document. Switching to greedy decoding and caching results per sentence brought full rechecks down to ~0.5 seconds.
The editor also had to feel native. I used range-based decorations and character-offset caret preservation so suggestions don't disrupt the cursor or existing formatting.
Finally, I worked through several Python 3.14 and Windows compatibility issues, including torchvision/protobuf imports, chat-template return shapes, BPE detokenization, and a llama-server IPv4/localhost quirk.
Accomplishments
Built a genuinely offline writing assistant that runs on a standard laptop. The GGUF model is just 762 MiB and generates at roughly 21 tokens/sec on CPU. Built two distinct workflows with a familiar writing experience: automatic grammar correction and on-demand paraphrasing. Kept all inference local, with no subscription or server dependency. What I learned Small models can be useful: A 1B Q4_K_M model is small and fast enough for interactive CPU use. Tiny models need defensive engineering: Structured output needs to be treated as unreliable rather than assumed to be valid. Speed is a systems problem: Debouncing, incremental processing, and caching had as much impact on responsiveness as the model itself.
What's next
More African language support. Better correction quality for longer and more complex writing.