FamiLyOs

FamiLyOs is a voice-first family COO that preserves context, catches risks and autonomously drives long-running tasks—from shared documents to a confirmed embassy appointment.

Comment

Inspiration

Family responsibilities rarely begin and end in one place. A document arrives on a phone, a decision is discussed by voice, another family member continues the task, and critical context gets lost across chats, inboxes, files, devices, and AI assistants.

That is the founder’s real pain: not the absence of another chatbot, but the difficulty of transferring a case “in the middle” without losing its history, documents, deadlines, decisions, or responsibility.

We created FamiLyOs as a voice-first family operating system—a family COO that preserves context, identifies risks, coordinates routine work, and keeps long-running operations moving until the family reaches a real-world result.

What it does

For this hackathon, we focused on one demanding Taskmaster scenario: organizing a multinational family trip from Madrid to the Philippines.

A shared invitation and an outcome-level voice instruction launch a Travel Case:

“Ly, organise this trip for our family. Handle everything you can and involve us only where a person is legally required.”

From there, Ly:

  1. creates and maintains one durable Travel Case;
  2. checks a disputed visa claim against an official-source fixture;
  3. assembles the required application package from versioned family documents;
  4. manages a bounded, time-compressed embassy communication thread;
  5. follows up when no response arrives;
  6. reacts when the embassy requests updated medical insurance;
  7. locates and validates the correct document version in the Family Vault;
  8. attaches the supplement without duplicating previous actions;
  9. continues until the appointment is confirmed without document blockers; and
  10. gives the applicant an originals checklist, document-review action, reminder, and preparation offer.

The family sees the entire operation in one mobile Control Room instead of reconstructing it from emails, files, and chat messages.

Why it is an agent, not a chatbot

The user gives Ly an outcome and scoped authority—not a sequence of prompts.

Ly maintains durable state, evaluates evidence, selects typed tools, follows up after a no-response window, handles new requests, retries safely, and stops at boundaries that legally or ethically require a person.

Success is not a generated answer. Success is a completed operational state: in our prototype, a confirmed embassy appointment with no outstanding document blocker.

Each execution stage also shows who initiated the action and who completed it, making the agent’s responsibility visible rather than hiding it behind a conversational interface.

How we built it

FamiLyOs combines:

  • a mobile PWA with Android Share Target;
  • Firebase Hosting and Firebase Authentication;
  • a FastAPI backend deployed on Google Cloud Run;
  • one bounded TravelOperationsAgent built with Google ADK 2.7.1;
  • Gemini 3.6 Flash through Vertex AI;
  • Cloud SQL for PostgreSQL as the canonical Case and execution state;
  • private Google Cloud Storage for versioned Family Vault documents;
  • Cloud Tasks for asynchronous intake processing;
  • durable agent-run and step checkpoints;
  • typed, idempotent tools with replay and cancellation protection; and
  • English and Ukrainian application interfaces.

The physical mobile Share flow is working on Android. The deployed Google ADK and Gemini runtime is validated through a dedicated Cloud Run preflight.

The complete embassy workflow uses deterministic synthetic adapters for the embassy, OVAS, bank, and email systems. It is deliberately time-compressed and clearly labelled in the interface. No real embassy messages, financial operations, or production personal data are used.

Challenges we faced

The first challenge was balancing autonomy with safety.

An agent that only drafts messages does not complete meaningful work. But an agent with unlimited permission to contact institutions, modify declarations, sign documents, or spend money would be unsafe.

We solved this by modelling authority as a bounded, revocable capability tied to one person, one Case, and one communication thread. Signatures, payments, changed declarations, certified translations, legal or medical consent, and physical attendance remain human-only actions.

The second challenge was reliable replay. Retrying a workflow must not create another Case, send a duplicate message, or attach the wrong document version. We introduced durable checkpoints, idempotency keys, immutable document versions, exact sent-document manifests, and fail-closed validation.

The third challenge was presenting a complex agentic workflow without turning the interface into an engineering dashboard. The Control Room focuses on the current outcome, completed actions, blockers, and the family’s next required step, while technical diagnostics remain secondary.

What we learned

The most important lesson was that useful agentic software needs a durable state model more than it needs a larger prompt.

A chat turn is temporary. A family Case has participants, evidence, provenance, deadlines, authority, ownership, and a definition of completion.

We also learned that families need continuity across people, devices, formats, files, and AI models. The supervising family agent should preserve that continuity while specialist agents remain replaceable and purpose-limited.

Finally, proactive assistance is most valuable before a problem becomes urgent: finding a visa-rule conflict, an expiring document, a missing insurance policy, or an unanswered embassy thread before the family gets blocked.

Existing work and hackathon contribution

FamilyOS was created on August 18, 2026, after the hackathon submission period opened. Its first iteration established the Case/Vault foundation and deployed mobile Share relay.

We then added the Google ADK and Gemini 3.6 agent runtime, durable agent runs and steps, travel requirements and document lineage, official-claim verification, the synthetic embassy workflow, the Travel Case Control Room, the bilingual FamiLyOs interface, and deployed proof endpoints.

What’s next

Our long-term goal is a voice-first operating system for family and personal responsibilities.

Gemini will act as the supervising family agent and delegate bounded tasks to specialist agents for legal, healthcare, research, engineering, and dispatch work. Cases will move safely between family members, devices, files, and AI models without losing context or forcing the family to start over.

FamiLyOs is not intended to replace family decisions. It is intended to remove the administrative friction surrounding them—and carry routine work all the way to a visible result.

Built With

Share this project:

Updates

Submission history