Sillon

Field reps send voice notes, typed lines, images, and photos of their notebooks. Sillon's AI agent fleet turns that into structured market intelligence in seconds, and fills spreasheets nightly.

Comment

Inspiration

I am a software engineer and AI developer. I do not work in distribution and I have never carried a sales bag.

This started as a conversation, not a project. A friend of mine works at a flour mill in Guinea, and he was telling me about the parts of his job that wear him down. Somewhere in the middle of it he described the spreadsheet.

The mill has more than thirty technico-commercial field reps visiting bakeries across the country. All day they send back what they find — voice notes, photos of flour reserves, typed lines, quick messages — into WhatsApp groups. Every day, one person in the back office works through all of it and retypes it into Excel.

Then, still complaining rather than explaining, he mentioned that he was already saturated by one rep's reports.

One out of thirty.

He said it as a grievance. I heard arithmetic. That is not a slow process, it is a wall — thirty field reps through one person is not thirty times harder, it is impossible. So most of what the field sees never survives the trip. What reaches the spreadsheet is whatever fits in a cell — a volume, a name, a date — from a fraction of the reps. A complaint that should have become a ticket with a diagnosis ends up as a line in a remarks column. Stock levels become a snapshot instead of a trend. The commercial director is not flying blind. He is steering on a thin, late slice of what his own team already knew.

The obvious project is a CRM. That is also the project that fails. He was clear about why: the reps are on bad connections, in a hurry, standing in someone's bakery. They will not fill in forms. Every digitization attempt in that world dies the same way — the tool becomes a surveillance instrument nobody uses, and everyone goes back to WhatsApp.

So Sillon does the opposite. It does not change what the reps do. It takes the retyping away from the person in the middle.

What it does

Reps keep working exactly as they do now. They send a voice note. They type a quick line — a price, a quantity, a name. They photograph the flour reserve. They photograph the page of the paper notebook they already carry. A visit is a bounded session: they start it, send whatever they want in whatever order and in whatever mix, and end it.

Then a fleet of AI agents does the rest, in about thirteen seconds (measured end to end on the deployed service):

  • Perception is a graph of Gemini specialists on ADK's Workflow runtime — one per channel: the French audio, the reserve photos, the handwritten page, commercial documents like a competitor's delivery note, product shots — running in parallel, with deterministic code reconciling their slices into one consolidated report. Counts come from what the rep said, typed or wrote; never from counting objects in a photo, because a camera in a doorway cannot see a whole storeroom. The photo does what words miss instead: yeast stored against the oven, a competitor's poster, damp sacks on bare ground. A brand printed on paper can never reach the stock numbers, because the paper-reading specialist is not allowed to produce them.
  • Technical triage diagnoses the baker's complaint — considering yeast, kneading, proving time, ambient humidity and oven calibration, not just the flour — then routes a ticket with an SLA.
  • Market intelligence recomputes share of shelf, detects competitor surges, and interprets figures the code calculated. The model never does arithmetic.
  • Memory rewrites a rolling digest of the bakery so the next visit starts with context.

The rep gets a card back in the chat thread with the counts, the diagnosis, the market movement, and any conflict the system noticed between its sources.

When it cannot read something, it asks — once. An unknown abbreviation on a handwritten page becomes one question with tappable answers. The answer teaches that rep's personal lexicon, amends the card, and is never asked of them again. When three reps independently confirm the same meaning, it becomes shared knowledge and stops being asked of anyone. And the answer is handled with the same honesty as the reading: name a brand the referentials do not carry and the sacks move to "Autre" with a note that a new brand is worth adding — never a silent guess onto a real competitor.

The thread works in both directions. A photo sent after the report — the retake the system asked for, or a page the rep forgot — produces an amended card that points at the original, which stays in place: the thread is the audit trail. Type a question mid-visit and an in-thread assistant answers it, grounded in that bakery's own history (ADK sessions with Vertex AI Memory Bank), scoped to the rep's own bakeries. And when the lab answers a routed ticket, the verdict travels back into the thread as "à dire au boulanger" — the loop closes on the person standing in the bakery, not in a dashboard nobody reads.

Every night, with nobody involved: a consolidation agent fills the spreadsheet the back office already uses — theirs, in their own column layout, with an agent working out what each column means — and writes a daily briefing. A tour planner reads the open alerts and builds each rep's next day, ordered by priority, grouped by zone, with a specific reason for every stop.

The back office stops retyping and starts the morning with all thirty field reps already structured.

How we built it

  • Gemini 3.5 Flash on Vertex AI for every model call, served from the global endpoint
  • Google ADK 2.8, used as a framework rather than a wrapper: perception is an ADK Workflow graph (conditional routing, per-node retries and timeouts, a deliberately higher thinking budget on the handwriting node alone) and is the deployed production engine — measured faster on Cloud Run than the single-call version it replaced; ADK plugins centralise output validation and stream per-agent cost/latency to BigQuery; two internal agents (lab, logistics) are served over the A2A protocol and the triage hand-off calls them; the assistant runs on ADK sessions with Vertex AI Memory Bank; and the fixture library doubles as an ADK evaluation dataset, including a hallucination-grounding metric on the triage agent
  • Cloud Run for the API and the worker, Pub/Sub decoupling capture from cognition, Firestore for state — with deployed security rules enforcing three-tier roles (agent / manager / admin) down to the document level — Cloud Storage for media, Cloud Scheduler driving the nightly jobs
  • Cloud Trace carrying an end-to-end span tree from the Pub/Sub push through every graph node, the fleet, and persistence
  • Flutter for both clients — a WhatsApp-shaped field app and a manager dashboard — sharing one codebase and two entrypoints. The field app runs as a PWA and builds as a native Android app from the same source

The architecture's load-bearing decision is that capture and cognition are decoupled. The client uploads media straight to Cloud Storage via signed URLs and posts metadata to an endpoint that returns 202 in under a second, then publishes to Pub/Sub. On a bad connection in Conakry, the message just sends. If Gemini is slow, if a photo takes twenty seconds, if a specialist agent fails — the person in the bakery never sees it, and Pub/Sub retries.

Clients capture, AI agents decide, only the server writes facts.

Challenges we ran into

Our test fixtures were lying to us. Through most of the build, the reference voice note claimed twelve sacks of one brand while the reference photo showed three other brands and none of it. Every result was read through that noise, and we could not tell a real defect from a bad fixture. We stopped and built a scenario library: twelve hand-authored situations with ground truth written before the pipeline ever ran, because expectations derived from output only ever assert "unchanged," never "correct." It found four real defects on its first run.

Then synthetic fixtures turned out to be too kind. AI-generated handwriting is regular, evenly spaced and reads like a font. One real page — written fast, photographed on a phone — exposed in about forty seconds what twelve generated ones had not.

Structured output made the model worse. Binding a response schema to the extraction call returned empty sack counts on a photo where the unconstrained call read them correctly. Reproduced twice under deliberate A/B. We validate with Pydantic afterwards instead.

The model's confidence score is not a signal. It never left 0.80–1.00 regardless of input quality — an unreadable page came back at 0.90. Anything routing on it would have been reading noise.

Latency is a product constraint, not an engineering metric. A rep is standing in a bakery waiting for a reply. Running the agents sequentially would have taken well over thirty seconds; the parallel graph keeps the whole measured chain near thirteen — and splitting perception into specialists turned out faster than the single big call, not slower, because each specialist reads only its own slice.

The deploy caught what a green test suite could not. Shipping was two-stage — a zero-traffic candidate revision measured first, then the traffic shift with live verification. The candidate caught the new image running out of memory before a single user saw it; the live checks caught one real defect that had walked through every green suite (a semantics change made outside the gated tests), and rollback to the previous revision took one command and about seven minutes of exposure. The fix shipped the same day, with the rule that prevents a repeat written into the project's context file as a rule, not a note.

Accomplishments that we're proud of

The system notices when its own sources disagree. When the voice note says one thing and the photograph shows another, Sillon does not silently pick a winner — it records both, prefers the rep who was physically present, and tells him what it saw. That behaviour was not designed in advance; it fell out of separating the sources honestly.

It asks once and never asks again. The clarification loop is the part that makes this an agent rather than a pipeline — it adapts to the person using it. It is dismissible without penalty, batched into a single question, and the silence on the second visit is the proof it worked.

A real handwritten page found a bug synthetic ones could not. Two different models independently read a closed o as a d — and the count still came out right, because raw transcription and expansion are separate layers. The same page also exposed a path where an unknown abbreviation became a real brand with four sacks attached to it. That one was a genuine data-corruption bug, and it only existed on real input.

It fills their spreadsheet, not ours. A mapping agent reads the customer's own column headers and proposes what each one means; a human confirms once. Nobody abandons the spreadsheet their business runs on.

About 2,000 automated checks across 22 suites, including a live verification chain that runs the whole thing end to end against the deployed service — sixteen hand-authored scenarios with ground truth written before the pipeline ever ran, deterministic suites for every rule the code owns, and rules-level checks that sign in as real Firebase users and probe the deployed security rules from outside.

What we learned

Photographs are bad at counting and good at noticing. The first version counted sacks from photos of the reserve and was consistently wrong, for a reason no prompt fixes. Reframing images as evidence rather than as a counting instrument made them more useful, not less: an observation about yeast stored beside an oven later became the evidence behind a technical diagnosis.

Adoption is a design constraint, not a rollout problem. The reason previous attempts failed was never the software. It was that they asked busy people on bad connections to fill in forms. Everything good about this system follows from refusing to do that.

Ground truth has to be written before the system runs. Otherwise your tests assert that nothing changed, which is not the same as asserting that anything is right.

Deterministic code should decide; models should interpret. Share-of-shelf arithmetic, ambiguity detection, illegibility measurement, promotion thresholds — all code. Diagnosis, interpretation and prose — the model. The failures we avoided were mostly on that boundary.

Frameworks encode assumptions worth checking. ADK threads session history into every call by default, which silently breaks a one-shot request. Nothing errors; results just drift.

What's next for Sillon

  • Market map — bakeries plotted and coloured by dominant brand, filterable by zone, rep and period, so share of shelf reads at a glance instead of bakery by bakery
  • Deeper offline support for rural zones — the outbox already queues messages and visit lifecycle events; full local-first sync with conflict resolution is the next step
  • Local languages — Susu, Pular, Malinké — which is where a voice-first design pays off most; the test library already carries a first code-switched Susu/French scenario, and the honest finding is that Susu survives transcription only re-spelled in French orthography today
  • Demand forecasting on accumulated visit data — once a year of real visits exists, the mill can plan production against what the field actually sees
  • Full multi-tenancy. The export configuration already lives under tenants/{id} and every write carries a tenant ID, but the other collections still sit at the root. Scoping them — and moving the referentials under the tenant — would let one deployment serve several mills, each with its own brands, taxonomy and spreadsheet.
  • Other verticals. Most of what makes Sillon flour-specific is configuration rather than code — brands, products, zones, taxonomy and thresholds are all editable. The domain reasoning would need rewriting: the triage agent knows about yeast and proving time, and a brewery would need its own. A template, not a drop-in. But the shape of the problem repeats across distribution in emerging markets — WhatsApp in, Excel out, one overwhelmed person in between.

Built With

  • bigquery
  • cloud-run
  • cloud-scheduler
  • cloud-storage
  • cloud-trace
  • dart
  • docker
  • elevenlabs
  • fastapi
  • firebase-auth
  • firestore
  • flutter
  • gemini
  • gemini-3.5-flash
  • google-adk
  • google-cloud
  • google-genai
  • google-sheets-api
  • multi-agent
  • multimodal
  • opentelemetry
  • pub-sub
  • pydantic
  • python
  • vertex-ai
Share this project:

Updates

Submission history