A voice-driven, context-aware daily planner: tell it your day, answer a couple of spoken questions, get a feasible plan with proofs — and change it by talking.
Inspiration
I walked out of the grocery store and remembered that the hardware store next door had what my other project needed. One more trip the next day, because nothing had looked at my day as a whole. Most people know their errands; the hard part is order, dependencies, which store, and whether a detour is even worth it. That planning load hits hardest for people with executive-function challenges, and it's a problem everyone recognizes on a messy day.
What it does
You tap a magnifying glass and describe your day out loud, in any order: "Pick up the kids at Lowell High School at three, I need groceries, get a haircut, dinner at home by seven." Magnified:
- reads your words into task cards and labels every detail by provenance — you said, estimate, guessed — confirm, needs you;
- asks only what changes the plan, by voice, with answers of a few words: "AM or PM?", "do you already have the ingredients?", "Where for the haircut? Say a barbershop's name, or say wherever fits my route";
- finds the real places: a named school resolves to the school (never a store); a "groceries" errand becomes candidate stores around every stop of your day, and the planner picks the one that fits the route — e.g. the store near the school while you'd otherwise wait for pickup;
- plans the whole day with an exact solver: fixed appointments, deadlines, opening hours, dependencies, optional stops with a value, waiting cost — and reads the plan back: "Here's your day: 4 stops, 38 minutes of driving. First… Left for another day…";
- explains with evidence: when it says a stop can wait, it re-solved your day with that stop forced in and measured the extra minutes; when a day is impossible, it tells you the single change that would make it feasible;
- adapts by voice during the day: "the dentist moved to 4:30", "running 20 minutes late", "skip the pharmacy", "what's next?" — each becomes a proposed change you confirm, then it re-solves from the actual time.
It runs on a laptop; the iPhone reaches it through a private, authenticated HTTPS gateway. Your tasks, plan and the language model stay on the laptop.
How I built it
The core idea is a division of labor: the AI listens, deterministic code verifies, an exact solver decides.
- Planner — Google OR-Tools CP-SAT. The day is a circuit over a virtual depot and task nodes (
AddCircuit); optional tasks may take a self-loop (skipped); a task with several candidate stores becomes a group of copies of which exactly one is visited, so the store is chosen as part of the route. Start/wait integers are propagated along chosen arcs with reified constraints; fixed starts, deadlines, opening windows and prerequisites are hard. The integer objective weighs driving fully, deferred optional value ("worth up to N extra minutes today"), waiting and time at optional stops at a reduced rate, and a weak earlier-finish term.
The objective the solver minimizes, in tenths of a travel minute (so every term is an integer):
$$ \min\; 10\,w_t \sum_{i,j} x_{ij}\,\tau_{ij} \;+\; 10 \sum_{t\in\text{optional}} v_t\,(1-p_t) \;+\; w_{\text{idle}}\Big(\sum_i W_i + \sum_{t\in\text{optional}} d_t\,p_t\Big) \;+\; T_{\text{finish}} $$
where $x_{ij}$ are the route arcs, $\tau_{ij}$ the travel minutes, $p_t$ the presence of an optional stop, $v_t$ its value ("worth up to $v_t$ extra minutes today"), $W_i$ the waiting before stop $i$, $d_t$ the time at the stop, and $w_{\text{idle}} \in {1, 3, 8}$ for less travel / balanced / fewer gaps. Times propagate along chosen arcs, $S_j = S_i + d_i + \tau_{ij} + W_j$ whenever $x_{ij}=1$; fixed starts, deadlines, opening hours and prerequisites ($S_t \ge S_p + d_p$) are hard constraints, never traded against the objective. A second solve locks the proven cost and minimizes deviation from the last accepted plan, so the same input always gives the same plan. Explanations come from counterfactual re-solves (each deferred stop forced in, each kept stop forced out, each constraint relaxed one at a time).
- Reading — Qwen3-4B locally via Ollama, 4-bit, JSON-schema-constrained decoding, thinking off. The prompt's first rule is "copy, don't compute": time fields hold the user's literal words; every model claim (optional? duration stated?) is checked against the user's own text; a regex time parser returns both AM/PM candidates for "2" and the app asks. If the model is down, a labeled rule-based extractor takes over.
- Voice — Cartesia (
ink-whisperSTT,sonic-3.6TTS) behind one server-side key. Recording is user-started and stops itself after a pause (RMS silence detection); audio is forwarded and discarded. The conversation is a deterministic server-side state machine; unclear speech is re-asked once with what was heard, never guessed. - Places and roads — Google Places API (New) and Routes API behind provider interfaces, with intent inference by whole-word matching, name/address validation of every candidate, and a labeled offline estimate as fallback. MapLibre + OSM by default; Google map with a referrer-restricted browser key.
- Stack: FastAPI + Pydantic v2, SQLite (aiosqlite), React 19 + TypeScript + Vite, Playwright browser suite, pytest. Phone access via a gateway instance (access code → HMAC-signed HttpOnly cookie, rate-limited) and an HTTPS tunnel; Ollama, dev servers and the database are never exposed.
Challenges I ran into
- A hallucination that mattered: "Lowell High School" once resolved to a hardware store — the hardware intent list contained the token "lowe" and matching was prefix-based, so the school "didn't fit" and a nearest-store fallback kicked in. Fix: whole-word intents, destination vocabulary suppresses shopping intents, and a provider result is attached only if it carries the name or address the user said. Regression tests pin it.
- Explaining deferrals honestly required the explanation to be in the same units the solver used — so each deferred stop reports the measured driving, waiting, time at the stop and later finish, and their cost against the value the user gave it.
- Determinism with a parallel solver: 8 workers return any of several tied optima; the tie-breaking second solve fixed plans that reshuffled for no reason.
- iPhone audio: WebKit refuses sound that doesn't begin inside a tap, and the assistant's reply begins after a network round-trip. A few milliseconds of silence played inside the tap on one shared audio element solved it (Chrome on iOS is WebKit too).
- Local model latency on CPU: Qwen3's chain-of-thought took minutes; turning thinking off, constraining output to a schema and bounding tokens made extraction practical.
Accomplishments that I'm proud of
- A complete voice loop — describe → clarify → plan → read back → correct — running on a phone against a laptop, with the plan produced by a proof-carrying solver.
- "Never guess" held end to end: AM/PM, unclear speech, unknown places and ambiguous task references all become questions.
- Route-aware store choice as part of the optimization, not a pre-step.
- 98 backend tests, a 74-check browser suite (83 with the real local model), and regression tests for every failure found on the device.
What I learned
- Pretrained models are excellent listeners and poor planners; the trust comes from where you stop using them. Grammar-constrained decoding plus verification against the user's words is far more reliable than prompting for schedules.
- An exact solver is not just more correct — it enables explanations by re-solving, which users understand better than any generated prose.
- Mobile web voice has real platform rules (gesture-gated audio, HTTPS-only microphone) that shape the UX.
What's next
- Duration estimates learned from the user's own history (on-device), multi-day planning, calendar import with explicit scopes, a self-hosted router for road times, and an accessibility pass (reduced motion, keyboard map alternatives).
Built With
- cartesia
- fastapi
- google-maps-platform
- google-or-tools
- google-places
- google-routes-api
- maplibre-gl
- ngrok
- ollama
- playwright
- powershell
- pydantic
- pytest
- python
- qwen3
- react
- sqlite
- typescript
- vite
Log in or sign up for Devpost to join the conversation.