SilverAssist AI

Inspiration

SilverAssist was inspired by a simple question: how can multimodal AI be genuinely useful for older adults in everyday life? Many seniors need help with practical tasks like checking the weather, understanding medicine packaging, and reaching a caregiver quickly in a stressful moment. We wanted to build something calm, supportive, and useful in those real daily situations instead of a generic chatbot demo.

What it does

SilverAssist is a multimodal assistant for older adults. It combines text, voice, image understanding, reminders, and emergency workflows in a single interface.

The current version can:

answer everyday questions in simple language
provide live voice interaction with Gemini Live
explain medicine packaging from uploaded images
check weather and offer practical clothing advice
create reminders from natural language
contact a caregiver by email after emergency confirmation

How we built it

We built SilverAssist as a full-stack web app with a Next.js frontend and a FastAPI backend.

The frontend provides an elder-friendly dashboard with profile setup, weather, reminders, voice controls, medicine image upload, and emergency confirmation flows. The backend exposes REST endpoints and a WebSocket endpoint for live camera analysis.

We used Gemini through the Google GenAI SDK for multiple capabilities:

Gemini text reasoning for general assistance
Gemini Live for real-time voice sessions
Gemini vision for medicine and image understanding
Gemini TTS for spoken responses

We also integrated external services and tools:

Open-Meteo for weather data
SMTP for caregiver alert emails
Google Cloud Run for cloud deployment

To keep the project lightweight and reproducible, the current version stores user profile data and reminders in browser local storage instead of a separate cloud database.

Challenges we ran into

One challenge was orchestration. This project is not a single model call. It combines text reasoning, live voice, image analysis, browser automation, TTS, reminder parsing and emergency escalation. Making all of those parts feel like one assistant required careful backend routing and clear UI flows.

Another challenge was designing for the right audience. A senior-facing assistant must be easy to understand, low-stress, and practical. That meant simplifying responses, using gentle language, and avoiding dense or technical interactions.

Deployment was also a challenge. Because the project includes both a frontend and a backend, plus Playwright inside the backend, we had to make sure the containers worked correctly with Google Cloud Run and that the frontend was configured to point at the deployed backend.

Accomplishments that we're proud of

We are proud that SilverAssist is a working multimodal product rather than just a concept. It successfully combines Gemini text, Gemini Live, Gemini vision, and Gemini TTS into one experience.

We are especially proud that it already supports:

real emergency-assist caregiver flows
medicine image explanation
reminder creation from natural speech
elder-friendly UI and language design
deployment to Google Cloud Run

Most importantly, we are proud that the project stays focused on real user value. The strongest accomplishment is not a single technical feature, but how those features come together to support older adults in everyday situations.

What we learned

We learned that the best AI experiences often come from solving concrete human problems. Features like medicine explanation, reminder creation, and caregiver escalation become much more meaningful when designed for a specific group of users.

We also learned that building multimodal AI products depends heavily on system design. Model quality matters, but so do prompt design, response style, frontend accessibility, backend orchestration, and deployment reliability.

Another important lesson was that tone and trust matter. For older adults, a calm and clear interaction style can be just as important as technical accuracy.

What's next for SilverAssist

The next step for SilverAssist is to make it more complete and production-ready. We would like to add persistent accounts, cloud-backed reminders, stronger caregiver workflows, and richer health support features.

Future directions include: