Baymax

A local-first health agent that connects records, wearables, and lived experience to understand what changes between clinical visits and proactively checks in with evidence, uncertainty, and consent.

Comment

Inspiration

A person living with a chronic condition may see a clinician four times a year.

Their condition does not disappear during the other 361 days.

Symptoms change. Sleep deteriorates. Medications are taken. Meals, stress, activity, and hydration vary. A wearable captures part of the story. The patient remembers another part. Clinical records hold a third and by the next appointment, those pieces are scattered across apps, portals, charts, and memory.

I built Baymax, shown in the prototype under the working codename Baymax, to make those days understandable.

What it does

The Baymax is a local-first, longitudinal health-intelligence system for people living with chronic conditions.

It connects:

  • clinical conditions, medications, encounters, laboratory results, and imported documents;
  • sleep, activity, recovery, heart rate, HRV, workouts, and measurements;
  • the patient's qualitative reports in their original words; and
  • deterministic personal baselines that reveal meaningful changes over time.

Instead of giving an AI model an unstructured pile of health data, the system separates three kinds of knowledge:

  1. Authoritative clinical state for exact facts such as active medications and conditions.
  2. Episodic memory for the patient's original experiences and conversations.
  3. Temporal intelligence for baselines, changes, and evidence-linked patterns.

A patient can ask a question or describe what happened today. GPT-5.6 calls typed tools to retrieve the appropriate clinical state, wearable comparison, timeline, imported documents, and relevant patient memories.

The resulting answer is schema-constrained. It must expose its evidence, uncertainties, and one safe action:

  • no_action
  • notify
  • check_in
  • escalate

The system does not diagnose, prescribe, or pretend to be a doctor. It observes, organizes, asks for missing context, and helps the patient prepare for the right human conversation.

The proactive loop

Baymax does not require the patient to remember to open a chatbot.

New symptoms, corrected narratives, clinical imports, wearable synchronization, and meaningful baseline changes can trigger a proactive evaluation.

The system may decide that no action is necessary, surface a low-risk observation, or ask a qualitative follow-up such as:

"Your recovery signals have shifted from your usual range. Did anything change in your sleep, stress, hydration, meals, activity, or medication timing?"

Approved check-ins first appear in the application. External outreach through iMessage is a separate transport and requires an active, purpose-specific consent grant immediately before delivery.

Every decision, consent check, tool call, and outbound message remains auditable.

Why this is different

Most health applications are dashboards, trackers, or chatbots.

Baymax is a continuously maintained, patient-owned health narrative.

It does not force every experience into a score. A statement such as:

"My pain was not terrible, but I stopped cooking because standing felt overwhelming."

may contain more useful information than a pain score alone.

The system preserves that statement as the authoritative source while deriving possible concepts such as fatigue, functional limitation, and activity change. Those interpretations remain correctable and are never silently converted into clinical facts.

It also compares each person against their own baseline rather than relying only on population averages.

How I built it

The product is a functional Expo application for iOS and Android connected to a local Hono coordinator.

The coordinator owns:

  • authentication and household isolation;
  • normalized health-data contracts;
  • the Vercel AI SDK agent;
  • model and privacy-boundary routing;
  • clinical and wearable provider connections;
  • longitudinal memory;
  • deterministic trend calculations;
  • proactive jobs and the durable outbox;
  • consent enforcement; and
  • audit records.

The prototype supports deterministic Synthea-compatible FHIR R4 records, imported FHIR bundles, clinical PDFs and text documents, Apple HealthKit, Android Health Connect, SMART-on-FHIR, and Open Wearables.

Patient narratives are written authoritatively to PostgreSQL before being asynchronously mirrored into Supermemory for semantic recall. If Supermemory is unavailable, exact episodes remain accessible through the local ledger and lexical fallback.

The same coordinator contract can run on a patient-controlled Mac, home server, clinic infrastructure, or hosted deployment. The mobile app can point to any of those environments without changing the product's domain model.

How GPT-5.6 is used

GPT-5.6 Terra is integrated through the Vercel AI SDK as the hosted reasoning route.

It does not calculate health trends or query the database directly. Instead, it calls narrow, typed tools for:

  • authoritative clinical state;
  • deterministic wearable trends;
  • the unified longitudinal timeline;
  • patient-authored check-ins;
  • semantic health-memory retrieval;
  • imported clinical documents;
  • proactive context; and
  • appointment preparation.

Exact calculations, dates, status, and evidence identifiers are produced by deterministic TypeScript code. GPT-5.6's job is to select the relevant evidence, communicate it clearly, preserve uncertainty, and determine an appropriate constrained action.

Every response discloses the model provider, model name, data boundary, tools used, latency, and whether a fallback occurred.

A local LM Studio model can run through the same OpenAI-compatible contract when the user selects strict-local inference.

How I collaborated with Codex

I built this project as an individual, with Codex as my primary engineering collaborator.

I directed the product vision and made the central decisions:

  • preserve the patient's original language;
  • separate authoritative clinical truth from semantic memory;
  • use deterministic personal baselines;
  • make every observation traceable to evidence;
  • expose model uncertainty and data boundaries;
  • keep external communication consent-gated; and
  • support both patient-controlled and hosted deployment.

Codex helped me transform those decisions into a working product.

Over one intensive build weekend, I used Codex to:

  • turn the initial concept into typed health-event, memory, consent, and agent-decision contracts;
  • inspect the Open Wearables and Supermemory implementations before designing their adapters;
  • build the Expo application and Hono coordinator;
  • implement the Vercel AI SDK agent and its tool-calling loop;
  • create deterministic clinical, wearable, and qualitative patient histories;
  • implement PostgreSQL persistence, idempotent ingestion, authentication, encryption, and background jobs;
  • connect HealthKit, Health Connect, FHIR imports, clinical documents, SMART-on-FHIR, and Open Wearables;
  • build proactive evaluation, quiet hours, snoozing, feedback, consent enforcement, and Sendblue messaging;
  • diagnose runtime and memory constraints and produce slimmer self-hosted profiles;
  • iterate on the native interface using simulator screenshots; and
  • continually test, review, document, commit, and integrate the system.

Codex did more than generate isolated code. It helped maintain architectural continuity while the project evolved from an idea into an end-to-end product.

The repository's dated commit history documents that build process during the hackathon period.

Challenges

The hardest problem was not connecting another health API. It was preventing different kinds of health information from collapsing into one unreliable model context.

A patient's recollection, a wearable observation, an inferred association, and an active medication are not equally authoritative.

I addressed this by separating structured clinical state, episodic memory, and temporal intelligence, while requiring provenance and stable evidence identifiers across all three.

The second challenge was local operation. HealthKit, mobile networking, model inference, semantic memory, databases, workers, and wearable services can overwhelm a development machine. I introduced fixture-only, local-product, and connected-provider runtime profiles so the system can remain useful without starting every optional integration.

Accomplishments

I am proud that this is not only a health chatbot mockup.

The repository contains:

  • a working native product experience;
  • a real GPT-5.6 tool-calling agent;
  • a durable longitudinal health ledger;
  • deterministic personal-baseline calculations;
  • evidence-linked qualitative memory;
  • proactive evaluation and outreach controls;
  • patient-controlled care-circle boundaries;
  • real provider connection interfaces;
  • synthetic but transparent and reproducible demo data; and
  • a portable local-first runtime.

Most importantly, the prototype demonstrates one complete loop:

records + wearables + lived experience
                    ↓
personal baseline and evidence-linked change
                    ↓
constrained GPT-5.6 decision
                    ↓
consent-checked app or iMessage check-in
                    ↓
patient response preserved in longitudinal memory

What I learned

The most useful health intelligence often begins with a person's own words, not another score. Local-first architecture is also not a binary claim. Storage, reasoning, messaging, and telephone communication each have different privacy boundaries, and the product must disclose those boundaries honestly. Finally, proactive AI should not mean autonomous AI. The valuable behavior is noticing, explaining, asking, and involving the appropriate human inside an explicit consent boundary.

What's next

The next stage is validation with real wearable accounts, physical-device HealthKit and Health Connect data, registered SMART-on-FHIR environments, and patient-caregiver households. I also plan to evaluate longitudinal-memory quality across Supermemory, PostgreSQL retrieval, pgvector, Mem0, and temporal graph approaches before committing to a production memory architecture.

The long-term goal is simple: Give people living with chronic conditions a patient-owned intelligence layer that remembers what happened, understands what changed, and helps the right human pay attention during the other 361 days.

Built With

Share this project:

Updates