Inspiration
Cloud writing assistants are everywhere, but they assume reliable, affordable internet. Across much of Africa, many people write on low-cost laptops with limited connectivity. Even with a connection, sending your writing to a remote server is a privacy tradeoff.
I wanted a writing assistant that works entirely on-device no internet, no subscription, and no data leaving the machine.
What it does
This is an offline writing assistant with two features in a Grammarly-style editor
Proofread: As you type, it checks grammar in the background and marks issues with inline underlines. A sidebar shows the original text, the suggested correction, and accept/dismiss controls. Paraphrase: Highlight a sentence and press Paraphrase to generate three alternative rewrites while preserving the meaning.
Everything runs locally. There are no network calls during inference.
How we built it
Frontend: Vite + React with a contentEditable editor, formatting toolbar, inline suggestions, an accept/dismiss sidebar, and a score indicator. Grammar checks are debounced and only rerun for sentences that changed.
Backend: FastAPI. Grammar correction uses CoEDIT, a FLAN-T5 editing model. I split text into sentences with wtpsplit, run each through CoEDIT, then generate a word-level diff and map corrections to character offsets for the editor.
Paraphrasing: Llama-3.2-1B-Instruct, converted to GGUF and quantized to Q4_K_M, runs through llama.cpp. A defensive parser converts imperfect model output into clean rewrite options.
Model tooling: llama.cpp's converter, llama-quantize, and llama-bench.
Challenges
I initially wanted CoEDIT as a GGUF model, but T5 generation doesn't work reliably in modern llama.cpp. The architecture loads, but llama-cli produces 0 tokens and llama-server hangs. I caught this early with a spike and switched to keeping CoEDIT as a regular model, while using Llama-3.2-1B for the GGUF/llama.cpp path.
The 1B model often produced malformed structured output, so I built a defensive parser that extracts valid results without letting bad JSON reach the UI.
CPU performance was another bottleneck. Beam search made grammar checks take ~40 seconds per document. Switching to greedy decoding and caching results per sentence brought full rechecks down to ~0.5 seconds.
The editor also had to feel native. I used range-based decorations and character-offset caret preservation so suggestions don't disrupt the cursor or existing formatting.
Finally, I worked through several Python 3.14 and Windows compatibility issues, including torchvision/protobuf imports, chat-template return shapes, BPE detokenization, and a llama-server IPv4/localhost quirk.
Accomplishments
Built a genuinely offline writing assistant that runs on a standard laptop. The GGUF model is just 762 MiB and generates at roughly 21 tokens/sec on CPU. Built two distinct workflows with a familiar writing experience: automatic grammar correction and on-demand paraphrasing. Kept all inference local, with no subscription or server dependency. What I learned Small models can be useful: A 1B Q4_K_M model is small and fast enough for interactive CPU use. Tiny models need defensive engineering: Structured output needs to be treated as unreliable rather than assumed to be valid. Speed is a systems problem: Debouncing, incremental processing, and caching had as much impact on responsiveness as the model itself.
What's next
More African language support. Better correction quality for longer and more complex writing.
Log in or sign up for Devpost to join the conversation.