LocalMind

• 

• 

• 

• 

• 

• 

• 

  Four specialists on one hub, each with its own persona, tools, and limits.

• 

  The approval gate. The agent has decided to write a file and stopped to ask first.

• 

  The operator dashboard. Join QR and code, the resident model, and live throughput.

• 

  A full agent run. It writes the daily figures, reads the file back to check its work, then asks again before writing the total.

• 

  Benchmarks run in public CI on Ubuntu 22.04, x86_64, CPU-only, which is the challenge's target platform.

• 

  One model is resident at a time under a hard RAM budget, so the hub can serve a 3B general model and an 8B Yoruba model on 8 GB.

Inspiration

A household, a clinic, or a farming cooperative in a low-connectivity area often has one shared computer and several people who need it. Cloud AI assumes bandwidth, a payment method, and an account. It also sends private data off-site in order to answer questions about it.

The machine is already in the room. It should be able to do the work.

What it does

LocalMind turns one computer into an offline AI hub.

The operator runs a directory of services. A service is a persona with its own tools, limits, and optionally its own model, layered over a local GGUF model. Everyone else joins from their phone over the local network by scanning a code. No accounts, no setup, no internet.

The demo hub runs four services on a single 8 GB laptop:

• Farm Advisor — crop storage and planting questions, offline
• Market Bookkeeper — an agent. It totals the week's sales and writes a file, but stops at an approval gate before it writes anything
• Health Helper — general guidance, with honest escalation to a clinic
• Yoruba Assistant — routed to a separate Yoruba model

Everyone is served concurrently, each with their own private history. Members can also create their own services from their phones.

How I built it

A Python monorepo: a FastAPI hub, llama.cpp with GGUF weights, WebSocket streaming, mDNS discovery, and a phone-first web client. A Tauri desktop app wraps the operator dashboard.

A service is a manifest persona, tools, policy, and an optional pinned model. The hub keeps one model resident at a time under a hard RAM budget and evicts on demand, so a hub can serve a 3B general model and an 8B Yoruba model without ever exceeding the budget.

The model was chosen by measurement. Candidates were benchmarked on x86_64 Ubuntu 22.04, CPU-only, in GitHub Actions, because my development machine is an Apple M3 and Metal numbers would not represent the target hardware.

Qwen2.5-3B-Instruct Q4_K_M was selected. Its accuracy sits inside the noise band of the 4B at n=100, and it is 28% faster and lighter. Speed carries 30% of the score and memory another 20%.

Challenges I ran into

The first speed numbers were wrong. 48 tok/s on the M3 was GPU-accelerated. The same model on CPU runs about 3.5 times slower. Every figure in the report is now CPU-only x86, measured in CI and reproducible from the repository.

There was a real path to an OOM, which is an automatic disqualification. The eviction logic would load a model larger than the entire RAM budget. An over-budget model is now refused up front with HTTP 507 rather than loaded and killed.

No general small model handles Yoruba. This was confirmed across four models: they loop, answer in the wrong language, or produce unusable output. Yoruba fine-tunes are fluent in conversation and unreliable on technical content.

The hub broke when taken offline. LAN address detection required a default route, so unplugging the internet made the hub advertise a loopback join URL that no phone on the network could reach. An offline-first product failing precisely when it went offline. Fixed.

Accomplishments that I'm proud of

The approval gate. Most on-device projects show a chatbot. This shows an agent that reasons, decides to write a file, and then stops and asks reaching the gate in under five seconds. The run stays paused and nothing happens until the person decides.

I also did not claim the African-language bonus. The available Yoruba models are not reliable enough to stand behind, so Yoruba ships as a real service on its own model and the bonus is left unclaimed, with the reasoning documented in the report.

What I learned

Measure on the hardware you will be judged on. Every assumption carried over from the Mac was wrong on CPU. And a benchmark that is not reproducible in CI is a story, not a number.

What's next

Validation on consumer laptops beyond CI runners, a 1.5B profile for 4 GB machines, and more local languages as usable fine-tunes appear.

Built With

• css
• docker
• fastapi
• gguf
• github-actions
• html
• javascript
• llama.cpp
• mdns
• python
• qwen2.5
• rust
• tauri
• websockets