Mallow — Project Story
Inspiration
Care work often disappears in the gap between doing something and being able to describe it as work.
A parent may remember the dentist appointment, compare two swimming classes, replace an empty bottle, follow up with a teacher, prepare everything for the next morning, and carry ten unfinished thoughts for somebody else. Much of this work has no clear start or finish, so timers and task managers do not capture it well. When it is not recorded, it becomes easy for everyone—including the person doing it—to believe that nothing happened.
I built Mallow for parents and guardians whose hands are already full. It does not require them to identify as a “primary caregiver,” meet a minimum amount of caregiving time, complete a form, or manage another productivity system. They can hold the rabbit, say one sentence, and leave.
Mallow gives that sentence a shape now—and returns later, on its own schedule, to say: I noticed.
What it does
Mallow is a mobile-first web app for recording household work, children’s care, and mental load through voice or text.
When someone speaks, Gemini extracts structured events while preserving the original words. Each event records:
- the exact supporting phrase from the transcript;
- the activity domain;
- the kind of labour;
- a duration or occurrence time only when the person actually stated one; and
- whether a value was asserted by the person or inferred by Mallow.
The model proposes the classification, but it does not decide the consequence. A deterministic Python policy maps labour_kind to a small response:
- invisible care and household maintenance become grass;
- mental load becomes a carrot;
- work that is already commonly recognised is recorded without food; and
- uncertain classifications are retained as unclassified without issuing a reward.
Grass, carrots, and leaves are not scores or goals. They are gentle visual forms for work that would otherwise leave no trace.
Mallow also has a second, autonomous loop. Cloud Scheduler wakes a protected Cloud Run endpoint. Mallow checks each workspace and decides whether:
- reflections are enabled;
- the person’s chosen reflection time has arrived; and
- something new has actually been recorded.
If any gate fails, Mallow stays silent. If all three pass, it builds a deterministic fact pack from that person’s records, asks Gemini to draft a short reflection, validates the result, and writes a folded leaf into the meadow.
There is no button in the app that creates a leaf. The browser cannot call the reflection endpoint. It accepts only an OIDC token issued for one Scheduler service account and one expected audience.
A chatbot responds when somebody is present. Mallow can decide to write back when nobody is there.
How we built it
Mallow runs on Google Cloud using:
- Gemini 3.7 Flash, accessed through the Gemini API in Vertex AI with the Google Gen AI SDK;
- Cloud Run for the Flask application and protected background endpoint;
- Cloud Firestore for workspace-scoped records, settings, garden state, and reflections;
- Firebase Authentication for Google sign-in and anonymous workspaces;
- Cloud Scheduler with OIDC authentication for autonomous reflection runs;
- Cloud Build and Artifact Registry for deployment.
The immediate capture path and autonomous reflection path are deliberately separated.
In the capture path, the browser sends voice or text to Cloud Run. The server verifies the Firebase identity, calls Gemini using a strict extraction schema, checks that every source_text is a continuous span of what the person actually said, applies the deterministic reward policy, and commits the result transactionally to Firestore.
Audio is used for extraction but is not persisted. The transcript is stored as spoken rather than rewritten or translated.
In the autonomous path, Scheduler enters through a different security boundary. It cannot impersonate a user, and a normal signed-in user cannot open the Scheduler door. Mallow assembles a fact pack only from stored records in the current workspace. A reflection that cites a nonexistent record, diagnoses the person, invents facts, or violates the output contract is discarded rather than repaired.
Records are content-immutable. Corrections, cancellations, and removals use explicit terminal states instead of silently overwriting history. Cancelled or superseded entries disappear from normal read surfaces while the underlying audit state remains internally consistent.
The repository includes a deterministic local model and labelled synthetic demo seeder so the full application and autonomous loop can be reproduced without exposing anybody’s personal records. The deployed product uses the real Gemini adapter; the deterministic model is never a silent fallback.
Challenges we ran into
Teaching the system the product’s point of view
One early classification rule treated “the school run” as already recognised work. That meant accompanying a child to school could be recorded without grass, even though child accompaniment is exactly the kind of care Mallow exists to make visible.
This was not an obscure edge case. It exposed a structural error: the activity’s surface label was overriding the caregiving relationship.
I separated activity_domain from labour_kind. A bus journey may belong to transport_errands, while still being invisible_chore when its purpose is accompanying a child. Conversely, a person’s own commute must not become childcare merely because Mallow’s users care for children.
The resulting semantic boundary was tested in both directions with real Gemini calls: explicit child accompaniment, natural shorthand such as “the school run,” bare personal journeys, code-mixed language, and numbers that look like times but are actually reference numbers.
Preserving the person’s account
A language model is useful because natural speech is messy. It is also dangerous if inference is allowed to overwrite what somebody said.
Mallow therefore keeps asserted and inferred information visibly separate. Durations are stored only when stated. source_text must be found verbatim inside the transcript. The model may classify the phrase, but it may not paraphrase evidence attributed to the person.
This changed my understanding of schema design: a schema is not merely an output format. It is a semantic gate that controls what the model is allowed to claim.
Making autonomy trustworthy
Running something every minute is easy. Deciding when not to act is harder.
The reflection loop normally produces nothing. A workspace that is not due, has reflections disabled, or contains no new records receives no leaf. When a draft fails deterministic validation, it is discarded.
I learned that autonomous behaviour is not demonstrated by how often an agent acts. Restraint, scoped memory, explicit eligibility rules, and reliable failure handling are equally important.
Building a complete identity lifecycle
Anonymous access initially looked like a simple onboarding shortcut, but it introduced real product questions: where the workspace lives, what follows a person to another device, how sign-out is confirmed, and what happens when someone later wants to use Google sign-in.
The final interface treats “Temporary workspace” as a state rather than a button. It explains that access belongs to the current browser and gives the person a clear exit before Google sign-in. Private navigation uses a signed, GET-only session cookie, while every write still requires a verified Firebase token.
This was a reminder that authentication is not plumbing hidden behind the product. It is part of the interaction architecture.
Making the mobile experience feel calm
The app had to work in real mobile conditions: iOS Safari, private browsing, microphone permissions, the on-screen keyboard, browser toolbars, cold loading, retries, and fingers that do not tap exact pixels.
Several important improvements came directly from physical-device QA: distinguishing typed “Say it again” from spoken “Say it again,” preventing the rabbit’s touch target from swallowing the leaf, allowing a pending capture to be cancelled without falsely claiming server success, separating status labels from buttons, and making the interface remain understandable when authentication fails.
The goal was not to add more controls. It was to remove moments where the person had to stop and wonder whether the app understood what they intended.
Accomplishments that we're proud of
- Mallow is deployed and working on Google Cloud. The production app runs on Cloud Run with Firestore, Firebase Authentication, Gemini through Vertex AI, and an OIDC-protected Cloud Scheduler job.
- The autonomous loop is real. In live testing, a workspace became due, Scheduler woke Mallow without a person pressing anything, and a new leaf appeared in the meadow. The browser cannot create that leaf itself.
- Natural language and deterministic policy are separated. Gemini interprets what happened; ordinary Python decides the consequence. A prompt change cannot silently redefine what receives grass or a carrot.
- The person’s words remain distinguishable from the model’s interpretation. Source spans must come directly from the transcript, while asserted and inferred information remain visibly labelled.
- The system has meaningful failure behaviour. Silence creates no request, quiet periods create no empty reflection, invalid notes are discarded rather than repaired, and cancellation cannot claim success before the server confirms it.
- The mobile interaction survived real-device QA. Voice, text, authentication, anonymous workspaces, cancellation, correction, time handling, records, and the autonomous reflection loop were repeatedly tested in iOS Safari—not only in a desktop simulator.
- The project is reproducible without private data. The public repository includes a deterministic local model, synthetic demo records, deployment instructions, an architecture diagram, and automated extraction, product, and browser tests.
The accomplishment I value most is smaller than the infrastructure: after recording many ordinary pieces of care, opening a leaf genuinely felt like being acknowledged. The technical system succeeded in producing the emotional experience it was designed for.
What we learned
The most important lesson was that an agent needs both probabilistic understanding and deterministic boundaries.
Gemini is responsible for interpreting natural language. Python is responsible for consequences, identity boundaries, state transitions, eligibility, and validation. Firestore holds durable state. Scheduler supplies time-based autonomy. None of those components can safely substitute for the others.
I also learned that product positioning must be compiled into the system—not merely written in a pitch. If the product serves parents and guardians, its prompt, schema, deterministic policy, substitute model, tests, and UI language must all understand care from that perspective.
Finally, I learned that acknowledgement does not need to become measurement. Mallow does not optimise a parent, rank their effort, or assign a daily target. It creates a small record, waits, and sometimes returns a leaf grounded in what they actually carried.
You say it once. Something else keeps the record—and, on its own time, tells you it noticed.
That is what being acknowledged feels like.
What's next for Mallow — A Gentle Agent for Invisible Labor
The next step is not to turn Mallow into a larger productivity system. It is to make the same promise more carefully and for more people.
The main future direction is a Protected Journal: a separate, private capture layer for words a person wants to keep even when they do not become a structured labour event. Today, Mallow’s model-facing schema is intentionally narrow. A journal layer would preserve the person’s fuller account without forcing every sentence through a labour classification or allowing a model inference to replace it.
Other next steps include:
- testing the care taxonomy longitudinally with parents and guardians from different household structures;
- expanding language and code-mix evaluation while preserving the original transcript;
- making care context and beneficiary more explicit in the structured schema;
- improving accessibility without turning the meadow into a dashboard;
- giving people clearer controls over retention, export, and deletion; and
- studying whether reflections remain helpful over months without becoming repetitive, diagnostic, or prescriptive.
Mallow should remain quiet by default. Its future is not more notifications, more scoring, or more demands. It is a stronger memory boundary, better understanding of care, and a gentler way to let invisible work leave evidence behind.
Log in or sign up for Devpost to join the conversation.