Comfort-z

An AI agent that watches, remembers, and notices when your animal’s behaviour changes.

Comment

Comfort-z 🐟

A persistent AI agent that watches for changes animals cannot tell us about.

Inspiration

Animals cannot tell us when something feels wrong.

That problem became personal to me after losing my betta fish, Raku. Looking back at individual moments was not enough. What mattered was the pattern across time: Was his posture changing? Was unusual behaviour persisting? Was the same concern appearing again?

Most visual AI tools answer a question about what they can see right now. I wanted to explore a different question:

What if an AI agent could keep a goal, observe an animal over time, remember what it had seen before, and decide when a pattern deserved attention?

That became Comfort-z.

Comfort-z is not designed to diagnose animals or replace veterinarians. It is designed to help owners notice meaningful behavioural changes earlier by turning observations into persistent, structured history.

What it does

An owner creates an animal profile and gives Comfort-z a monitoring goal such as:

"Keep an eye on Raku."

Comfort-z can then operate through bounded monitoring windows instead of requiring the owner to repeatedly prompt an AI.

During a monitoring window, it:

  1. Loads the animal's persistent monitoring profile and previous state.
  2. Retrieves the configured visual source.
  3. Samples selected frames rather than sending every frame to an AI model.
  4. Uses Gemini 3.5 Flash to determine whether the expected animal is sufficiently visible and to create a structured, non-diagnostic behavioural observation.
  5. Saves that observation to persistent memory.
  6. Retrieves meaningful history for the same animal.
  7. Compares the current observation with previous observations.
  8. Determines a trend and applies an explainable alert policy.
  9. Adjusts monitoring state for later bounded invocations.
  10. Produces a history that can later be summarized in a daily monitoring report.

Frames where the expected animal is not visible or visibility is uncertain are retained for provenance but excluded from behavioural trend persistence.

Comfort-z can also incorporate owner-provided direct readings and care updates. For example, an aquarium water-temperature reading is stored as owner-reported context rather than pretending it came from a sensor.

Optional outdoor weather from Open-Meteo is kept separate from direct enclosure measurements. Comfort-z is explicitly instructed not to assume that outdoor temperature equals aquarium, terrarium, cage, or indoor conditions.

More than a chatbot

Comfort-z is deliberately not:

prompt → model → answer

The persistent workflow is closer to:

goal → observe → remember → compare → decide → continue

The owner establishes the goal, while bounded invocations can continue the monitoring workflow using saved state.

This is why Comfort-z uses a persistent monitoring profile containing its source cursor, active state, sampling mode, daily sample budget, report schedule, and animal context.

Cloud Scheduler can invoke finite monitoring windows and report generation without keeping a Cloud Run process permanently alive. The deployed scheduler jobs are intentionally pausable so quota-controlled demonstrations do not consume model calls unnecessarily.

Architecture

The hackathon deployment uses Google Cloud Run as the application runtime and Google ADK for the agent/tool architecture.

A private video source can be stored in Google Cloud Storage. During a bounded monitoring operation, Comfort-z accesses the configured object using its Cloud Run runtime identity and samples selected frames with OpenCV.

Those frames are interpreted by Gemini 3.5 Flash, the primary multimodal reasoning model.

Structured observations are persisted in Cloud Firestore, allowing later monitoring windows to retrieve the animal's history and make comparisons across time rather than treating every observation as an isolated request.

The deployed service also uses Secret Manager for the Gemini Developer API credential.

The dashboard provides a read-only reference preview for configured monitoring videos. Preview playback and Gemini sampling are intentionally independent; the UI does not pretend that the frame currently playing is the exact frame being analyzed.

Conditional research

Comfort-z also contains a bounded research branch for concerning, worsening, recurring, or unresolved patterns.

When enabled, this branch uses Gemini 2.5 Flash with Google Search grounding as an auxiliary research model. Research context cannot override the monitoring decision, and failure does not interrupt the core monitoring workflow.

During deployment validation, Google Search grounding was unavailable under the project's current quota/access. Instead of hiding this, Comfort-z handles it as a safe research failure: the existing observation, history comparison, trend, and alert decision continue unchanged.

The primary monitoring workflow remains powered by Gemini 3.5 Flash.

Demo

For a controlled and reproducible demonstration, Comfort-z uses prerecorded footage of Raku as its monitoring input.

The footage itself is prerecorded; the agent processing is not presented as a live camera feed.

The deployed workflow retrieves the private source, samples it, interprets selected frames with Gemini 3.5 Flash, retrieves Firestore history, compares observations, and persists the resulting monitoring state.

In the demonstrated history, repeated unusual posture develops into a persistent concerning pattern, allowing Comfort-z to surface an explainable alert rather than treating each frame independently.

How we built it

Comfort-z is built primarily with:

  • Python
  • Google Agent Development Kit (ADK)
  • Gemini 3.5 Flash
  • Gemini 2.5 Flash for optional conditional research
  • Google Cloud Run
  • Cloud Firestore
  • Google Cloud Storage
  • Cloud Scheduler
  • Google Secret Manager
  • OpenCV
  • FastAPI / Uvicorn
  • Open-Meteo
  • Vanilla HTML, CSS, and JavaScript

The monitoring system was intentionally designed around bounded operations.

Rather than continuously sending video frames to Gemini, each invocation has explicit limits such as sampling intervals, maximum samples, retry bounds, and a daily sample budget. This makes the architecture more controllable and practical for an asynchronous cloud agent.

Challenges

One of the biggest challenges was designing "continuous monitoring" without building an uncontrolled infinite process.

The solution was to persist the monitoring goal and cursor while making each actual execution finite. Cloud Scheduler can trigger another bounded window later, allowing the agent to continue its goal across invocations.

Another challenge was distinguishing absence of evidence from evidence of normal behaviour. If the expected animal cannot be seen clearly, Comfort-z should not interpret a random object or empty aquarium as the animal. Visibility therefore became an explicit part of the observation model.

Persistent memory introduced another challenge: historical records should inform decisions without allowing uncertain frames to create false trends. Comfort-z filters behavioural history accordingly before comparison.

Cloud deployment also required carefully separating private media access, application state, credentials, and user-facing preview behaviour. The monitoring source remains private and the preview endpoint streams it through the authenticated application rather than creating a public or signed URL.

Finally, external research demonstrated why graceful degradation matters. Search grounding was not reliably available within the project's current quota/access, so research was designed as optional context rather than a dependency for the monitoring and alert workflow.

What we learned

Building Comfort-z changed how I think about AI agents.

The difficult part was not simply asking a model to analyze an image. The difficult part was deciding:

  • what should become memory,
  • which history is meaningful,
  • when the agent should act,
  • when it should do nothing,
  • how long an autonomous operation should run,
  • what happens when a model or external service fails,
  • and how to make those decisions understandable to the owner.

I also learned that persistence fundamentally changes an AI system. A single observation may mean very little. Several related observations across time can mean something very different.

That is the capability Comfort-z is exploring.

Accomplishments

I am especially proud that Comfort-z grew from a simple visual-monitoring idea into a deployed persistent agent workflow.

It can maintain an animal-specific goal, process bounded monitoring windows, distinguish valid observations from visibility failures, retrieve historical memory, compare changes, persist trends and alert decisions, incorporate owner context, generate reports, and continue operating across separate cloud invocations.

Most importantly, it remains explicit about what it does not know.

Comfort-z does not diagnose an animal. It does not fabricate sensor readings. It does not call prerecorded footage a live feed. And optional research failure does not become fabricated evidence.

What's next

The next step is moving beyond controlled prerecorded input toward durable owner-uploaded media and authenticated live camera or edge-device integrations.

Future versions could also add real notification delivery, hardware sensor integrations for enclosure measurements, richer multi-animal support, and stronger veterinarian-reviewed behavioural policies.

The long-term idea is simple:

Give animals a persistent digital observer that helps their humans notice when something changes.

Built With

Share this project:

Updates

Submission history