Carely

Inspiration

Carely began with a familiar problem: an older parent may know how to use a button phone but have no reliable internet access, no confidence with apps, and no direct way to use tools such as Gemini. They can call their family, but they may hesitate because the question feels too small or they do not want to interrupt someone at work.

We wanted to give them another number they could call—one that could answer patiently, remember trusted family instructions, and take useful actions without requiring a smartphone, an account, or a new interface.

What it does

Carely is a family-grounded voice agent for older adults.

The older adult calls from an ordinary phone and speaks naturally. Carely identifies the correct care recipient, retrieves relevant family context when needed, uses current public information when appropriate, and responds in clear, patient language.

Family members use a web dashboard to:

Register care recipients and trusted contacts.
Add household instructions, routines, preferences, and appliance guides.
Upload documents, photographs, audio notes, and short instructional videos.
Create and manage reminders.
Test Carely using text or browser voice.
Review transcripts, retrieved sources, completed actions, summaries, and conversation-quality scores.

Carely can also create a reminder during a conversation. It gathers the required information, repeats the complete reminder, asks for confirmation, and invokes the tool only after the caller agrees.

How we built it

Carely consists of two services deployed on Google Cloud Run.

The frontend is a Next.js application that provides authentication, family configuration, guide management, reminders, testing interfaces, and conversation history. The backend is a Hono service that runs the agent system, retrieval pipeline, multimodal processing, voice sessions, and tool execution.

Google ADK orchestrates the text and live voice agents. Gemini 3.5 Flash Lite provides reasoning and tool selection, while Gemini Live handles streamed speech, turn-taking, interruption, transcription, and spoken responses.

Each family receives an isolated Gemini File Search store. A deterministic retrieval gate decides whether a request requires private family context. Greetings and general questions skip retrieval, while questions about routines, medicines, preferences, relationships, or household devices can search the family’s saved information.

Gemini’s multimodal capabilities convert uploaded photographs, documents, audio notes, and short videos into searchable household guidance. Google Search is available for questions requiring current public information.

Deterministic application code remains responsible for identity, authorization, validation, emergency handling, prompt-injection boundaries, confirmation, and idempotent side effects. The model supplies intelligence, but it does not receive unrestricted authority.

Both services are deployed in the carely-505510 Google Cloud project. The deployed API uses Vertex AI, and its Cloud Run logs show Google ADK requests to gemini-3.5-flash-lite with backend: VERTEX_AI.

Challenges we ran into

The first challenge was making private family context useful without putting every document into every prompt. That would have been expensive, noisy, and unsafe. We addressed this with isolated File Search stores, source metadata, bounded retrieval results, and a deterministic gate that avoids retrieval when it is unnecessary.

Caller identity was another important boundary. The model must never decide which family owns a phone number. Carely validates the provider request, normalizes the caller’s number, resolves it to one saved care recipient, and creates a short-lived session before Gemini receives any private context.

Tool execution also required more than a prompt. Reminder creation needed explicit confirmation, authenticated internal requests, input validation, and database-level uniqueness so concurrent scheduler runs could not place the same reminder call twice.

Live voice introduced a separate systems challenge because telephone audio and Gemini Live use different formats and sample rates. Carely validates and converts the audio in both directions while preserving interruption and turn-taking behavior.

Deployment exposed an important infrastructure limitation. Cloud Run can host the application, but its writable filesystem is ephemeral. Our demo therefore runs one instance of each service. A multi-instance production deployment will require shared durable storage and distributed coordination.

Accomplishments that we're proud of

We built a complete agent system around an interface that already feels familiar to the intended user: making a phone call.

Carely combines:

Gemini 3.5 Flash Lite through Vertex AI.
Gemini Live for natural voice interaction.
Google ADK for agent orchestration and tools.
Gemini File Search for isolated family memory.
Multimodal processing for images, documents, audio, and video.
Google Search for current public information.
Confirmed reminder creation with idempotent delivery.
Deterministic identity, safety, and prompt-injection boundaries.
A family dashboard with guides, reminders, transcripts, sources, actions, and conversation reviews.
Two publicly accessible Cloud Run services built through Google Cloud Build.

The repository contains 66 automated tests covering retrieval routing, owner isolation, audio conversion, session reuse, signature validation, guide processing, prompt boundaries, reminder scheduling, conversation scoring, and failure handling.

Most importantly, the product does not ask an older adult to change their habits before receiving help.

What we learned

Accessibility does not always mean simplifying a screen. Sometimes it means removing the screen entirely.

We also learned that a dependable agent is defined as much by its boundaries as by its model. Identity must be established before generation. Retrieval must be conditional. Missing information must remain missing. Side effects must be confirmed, authorized, and idempotent.

Gemini was most valuable as a connected system rather than a single text endpoint. Gemini reasoning, Gemini Live, File Search, multimodal understanding, Google Search, and Google ADK allowed us to create one path from family-provided evidence to a grounded spoken answer.

Finally, deployment truth matters. A successful demo should clearly distinguish what is running on Google Cloud from what still requires durable production infrastructure or external provider configuration.

What's next for Carely

The next step is replacing local SQLite databases and uploaded-file storage with shared durable Google Cloud services. This will allow Carely to run safely across multiple Cloud Run instances without losing records or duplicating reminder delivery.

We also plan to:

Add stronger operational observability for agent decisions, retrieval, latency, and tool execution.
Expand multilingual and mixed-language conversations, including Hindi and Hinglish.
Improve family feedback so corrections can become better trusted context.
Add more carefully bounded tools for appointments, household coordination, and recurring routines.
Improve accessibility testing with older adults and caregivers.
Strengthen emergency escalation without turning ordinary questions into false alarms.
Complete broader live-carrier testing across different devices and network conditions.

Our long-term goal is simple: anyone who can make a phone call should be able to access a capable, personal, and dependable AI agent.