Pocket Buddy

Frost orchestrates wearable and mobile Skills to turn meals, movement, routes, and everyday discoveries into safe, traceable actions.

Comment

Inspiration

The name Frost comes from American science-fiction writer Roger Zelazny's 1966 story For a Breath I Tarry. In the story, Frost is a machine that seeks not only to calculate the world, but to understand human experience. That became Pocket Buddy's guiding question: what if an AI companion did more than answer once—what if it stayed with a person through the small actions that make up a day, understood the goal behind them, and returned visible, verifiable results?

Pocket Buddy turns that literary idea into an agent system. Frost is one consistent companion across software and an ESP32-S3 round-screen badge. The software holds the day; the badge stays close. Within Frost's Taskmaster loop, goal understanding, capability selection, controlled Skill execution, user confirmation, and evidence are separate responsibilities rather than one unbounded model call.

What it does

One day with Frost

At 08:10, breakfast, the user takes or chooses a meal photo. Photos proposes candidates and nutrition ranges, but the user reviews and confirms what was actually eaten. Only that confirmed summary can enter today's memory; the raw photo stays in its original workflow.

At 10:30, training, Frost chooses a next step from equipped and authorized capabilities. The phone provides the detailed exercise experience while the badge carries the essential cue close to the body. Frost can propose; the user remains the decision-maker.

At 18:20, a run, Earth / Routes uses AMap for real roads and the phone's current GPS. The full map and route stay on the phone while the badge gives lightweight prompts, so attention can remain on the road. Deviations trigger recalculation rather than pretending that the original plan is still correct.

At 18:45, a bird call, a long press on the badge captures a short nearby sound. Bird Listener validates the input, returns a candidate instead of false certainty, and brings the result back to the same round display and the phone's discovery record.

At 21:30, review, the day's confirmed meals, movement, route progress, and nature discoveries become a coherent review. Frost carries context forward without turning every photo, recording, health detail, or precise location into general cloud memory.

Across that day, Pocket Buddy brings together:

  • Photos for meal candidates, correction, and explicit confirmation.
  • Earth / Routes for real-road planning and route display through AMap.
  • Agents / My Skills for Frost, route planning, Bird Listener, Her Motion, and exercise experiences.
  • Skill Canvas for creating, compiling, previewing, and saving custom Skill graphs. It is an authoring surface; production execution remains behind the Taskmaster's registered-capability and evidence boundary.
  • Pocket Buddy badge for display, buttons, microphone, speaker, touch input, and a BLE phone bridge.

Frost does not simply generate advice. The server-owned Taskmaster validates structured model output, applies approval and permission boundaries, calls only registered capabilities, and records privacy-bounded execution evidence.

How we built it

The client is built with React, TypeScript, Vite, Capacitor, and native iOS bridges. The physical companion uses an ESP32-S3 badge connected to the phone through Bluetooth Low Energy. The phone and web interfaces share the same Frost identity and Skill vocabulary.

For the deployed agent workflow, the backend runs on Google Cloud Run. A server-owned Prompt Harness controls authority, context budget, task profiles, structured JSON validation, cancellation, and bounded continuation. It calls Gemini 3.5 Flash through the official Google GenAI SDK (@google/genai) using Vertex AI.

Every successful agent run receives a trace ID. Firestore stores restricted evidence metadata such as task type, model, status, timing, Cloud Run revision, and prompt profile. It does not store prompts, model responses, credentials, raw health content, recordings, photos, or precise routes. The deployed service requires Firestore evidence rather than silently reporting success when evidence storage fails.

The production deployment also uses Cloud Build, Artifact Registry, Cloud Logging, and a dedicated least-privilege runtime service account limited to the permissions needed for Vertex AI and Firestore.

Challenges we ran into

The hardest problem was not calling a model; it was defining where the model must stop. A generated plan is not a completed workout, a guessed meal is not an eaten meal, and opening a page is not evidence that a device action succeeded. We built explicit confirmation, timeout, validation, provenance, and evidence boundaries around those transitions.

We also had to connect a web app, native iOS capabilities, BLE hardware, external map infrastructure, and Google Cloud without turning every service into one unsafe data pool. The result is deliberately layered: Google Cloud owns agent reasoning and traceable execution evidence; AMap owns mainland route presentation; the phone owns sensitive permissions and device bridging; the user owns confirmation.

Accomplishments that we're proud of

  • Deployed a working Frost Taskmaster on Cloud Run in asia-east1.
  • Verified Gemini 3.5 Flash through Vertex AI and the Google GenAI SDK.
  • Required a matching Firestore evidence record for real agent runs.
  • Kept prompts, health details, media, credentials, and precise location out of the evidence store.
  • Connected one Frost identity across software Skills and a physical badge.
  • Preserved honest uncertainty and user confirmation in food, health, route, motion, and nature workflows.
  • Turned a list of features into one continuous, human-scale day with Frost.

What we learned

A trustworthy agent needs more than a good system prompt. It needs capability contracts, explicit authority, structured outputs, cancellation, least-privilege infrastructure, and evidence that can distinguish proposed, started, waiting, failed, and completed work. Physical interaction makes those distinctions even more important.

We also learned that continuity does not require collecting everything. The useful memory is often a confirmed summary and a traceable outcome, not the raw photo, recording, route, or private conversation that produced it.

What's next for Pocket Buddy

Next, we will connect more user-authored Skill Canvas graphs to the production Taskmaster only after each capability has explicit permissions, failure handling, and evidence requirements. We also plan to expand device testing, user-owned connectors, and privacy-preserving long-term memory while keeping raw media and sensitive context out of general cloud traces.

Pre-existing work disclosure

Pocket Buddy builds on earlier interface design, iOS/hardware integration, and product experiments. The team subsequently developed and integrated the Frost Taskmaster workflow, the Gemini 3.5 / Google GenAI SDK backend, the server-owned Prompt Harness, the Cloud Run deployment, Firestore evidence contracts, service boundaries, testing, documentation, and the demonstration experience presented here.

Team contribution

Cheng Zhang — I led product design and end-to-end implementation across the web/iOS experience, Frost Taskmaster workflows, wearable integration, privacy boundaries, and the final demo.

Try it

Built With

Share this project:

Updates

Submission history