Tilly — a farm memory you keep by talking

Smallholder farmers say what they did, in their own words. Tilly turns it into trusted, searchable farm records and answers from their own history — no forms, no notebooks.

Comment

Inspiration

Smallholder farmers do the work, but the history lives in their heads, in notebooks, and in scattered messages. When it's time to decide what to do next — when did I last spray? did I fertilise the north field? what did I do this week? — the answer is hard to find. Missing records mean repeated work, missed follow-ups, and guesswork.

Farm-management apps exist, but they ask a busy farmer to fill in forms. We wanted the opposite: record the farm the way farmers already talk about it, and get answers back from their own history.

What it does

Tilly is a farm memory you can talk to, in a phone or desktop browser.

  • Record by talking. Say "I sprayed the tomatoes in the north field this morning" and Tilly saves a structured record — activity, crop, field, date, time of day — and shows a receipt with Undo.
  • A spoken conversation, not a form. Tap the mic and talk: Tilly sends what you said when you pause, answers out loud, and reopens the mic for your reply. A hands-free mode listens for a wake word ("Hey Tilly, …").
  • Ask once, never guess. If an important detail is missing ("I weeded this morning"), Tilly asks one short question, then saves. If the farmer answers "I don't know" or just "thank you", Tilly saves the note with the gap, says so, and fills it in whenever the farmer mentions it later.
  • Remembers the conversation. "I sprayed there too" resolves against what was just said — but a crop or field is only accepted if the farmer said it or it already exists on the farm, so Tilly's own words can never invent a name.
  • Ask your own history. "When did I last spray?" or "What did I do this week?" gets a short answer that cites the exact records it used.
  • Correct by talking. "No, it was the back field" or "delete that" changes or removes a record, with an audit trail of every change.
  • Cited farming guidance. General questions like "How do I control aphids on tomatoes?" are answered from trusted agricultural sources (FAO, CABI, university extension services) found with Tavily, shown separately from the farmer's own records, with real links and a reminder that it's guidance, not a diagnosis.
  • Try it in one tap. The welcome screen offers example messages and a sample farm with three weeks of records, so anyone can reach the "aha" moment in seconds.

How we built it

NVIDIA Nemotron on Nebius Token Factory does the language work, in two roles:

  • NVIDIA-Nemotron-3-Nano-30B-A3B reads every message into strict JSON: intent, events, dates, fields, and question filters. It is fast and cheap, and is asked again with thinking mode only when the quick read looks doubtful.
  • Nemotron-3-Ultra-550b-a55b writes short, plain-language answers and reports — using only the records we hand it.

The model proposes; code decides. A value $v$ from the model becomes farm memory only if

$$ \text{save}(v) \iff \text{schema}(v) \;\land\; \text{policy}(v) \;\land\; \text{grounded}(v) \;\land\; \text{owned}(v) $$

— it passes Zod schema validation, the required-detail and duplicate policy, a check that any reference to the past ("same field as last week") matches a real confirmed record, and a farm-ownership check. Answers may only cite records the database actually returned.

Reports are counted by SQL, not by the model. PostgreSQL filters and groups the records; Nemotron only phrases the result, so the numbers are right.

Tavily searches a fixed list of trusted agricultural domains. Only the farmer's question is sent — never farm names, fields, or records. The model refers to sources by number, and code builds the links from Tavily's own results, so a URL can't be invented.

Deepgram gives Tilly a voice. The mic streams live to Nova-3 speech-to-text, which detects when the farmer has finished speaking, and replies are spoken with an Aura-2 voice. The Deepgram key never reaches the browser: the server issues a 60-second token for each conversation. Without Deepgram, Tilly falls back to the browser's own speech services.

Stack: Next.js 15 and React 19, PostgreSQL on Neon with Drizzle ORM, Zod, deployed on Vercel. Nearly 600 automated tests (with an in-memory Postgres) and a secrets scan run on every pull request.

Challenges we ran into

  • Trusting nothing without annoying the farmer. An early version asked too many questions. We settled on ask once, save always: at most one question per note; anything still missing is saved as a gap, never guessed.
  • Models drop details unpredictably. Live testing against the real model found cases unit tests couldn't: with farm memory in the prompt, Nemotron sometimes returned a question with no filter, so "When did I last spray?" listed the whole month; another time it asked "was it today?" after the farmer had said "this morning." Each fix was a small deterministic check plus a regression test — not more prompting.
  • A trust bug hiding in plain sight. Answering "which crop?" with "thank you" once saved "thank you" as the crop — and the receipt didn't show crops, so nobody saw it. Now non-answers are recognised in code without a model call, filler values are dropped, and receipts always show the crop.
  • Natural voice is hard. Token Factory has no speech models on our account, so we added Deepgram for speech in and out. Deciding when a farmer has finished speaking is still imperfect: Tilly sometimes cuts a farmer off mid-sentence. Interrupting Tilly while she speaks isn't supported yet either.
  • Keeping two sources of truth apart. Mixing web guidance into farm records would quietly destroy trust, so guidance is labelled, cited, and never saved as a farm fact.

Accomplishments that we're proud of

  • A first-time user goes from opening the link to a saved, structured record in one tap.
  • Every answer is grounded and cited: farm answers cite real records; guidance cites real pages.
  • A real, tested trust boundary: the model interprets language, but only code decides what becomes farm memory.
  • A farmer can talk to Tilly and hear it answer, and nothing they say becomes a record unless code can stand behind it.
  • Live, deployed, and usable in a phone browser.

What we learned

  • The most valuable job for an LLM here is interpretation, not storage or arithmetic. Let SQL count and code verify.
  • Testing against the live model finds failure modes that mocks never will. Every one we found became a small guard and a regression test.
  • Small UX details decide whether someone reaches the "aha" moment — like never asking for information the user already gave.
  • Voice changes everything about pacing: knowing when a person has finished matters as much as understanding what they said.

What's next for Tilly — a farm memory you keep by talking

  • Better listening: tune end-of-speech detection so Tilly never cuts a farmer off, let farmers interrupt Tilly mid-reply, and test with real farmers in noisy fields.
  • Timed reminders that reach the farmer — "remind me to weed tomorrow at 7am", delivered by phone notification or SMS even when Tilly is closed.
  • Install Tilly as a phone app, tested on low-end Android phones.
  • Real accounts and cross-device sync (the demo uses anonymous, browser-scoped farms).
  • Quantity-aware reports: litres, kilograms, costs, and yields.
  • Local languages, and voice that works on low-end phones.
  • WhatsApp and SMS as ways to talk to Tilly.
  • An evaluation set of real farmer messages to measure accuracy over time, and pilots with farmer cooperatives and extension officers.

Built With

Share this project:

Updates

Submission history