Inspiration

Every summer in Phoenix, older adults die alone at home in the heat, and cold snaps do the same across the country. Most of them had someone who would have come, if only that person had known in time. Texts and apps don't reach someone who is 82, confused by heat, and not looking at a phone. A phone call does.

What it does

Kinwell calls people on dangerous-weather days, has a real conversation, and gets help moving.

  • Family enrolls a parent on the Family site. A Gemini assistant asks one thing at a time (name, number, address, language, backup contact) by typing or voice, and validates each answer.
  • Permission first. Kinwell calls the parent and asks if it's okay to check in. Nothing starts until they say yes.
  • Weather-triggered calls. Every 30 minutes a risk engine checks National Weather Service alerts for every enrolled person's zone. On an Extreme Heat or Cold Warning, Kinwell calls.
  • A natural check-in in English or Spanish. Amazon Nova 2 Sonic, speech to speech in real time: Is the AC (or heat) working? How hot or cold is it inside? Water? Dizziness, confusion? For cold: power and gas, and is anyone using an oven or generator for warmth (carbon monoxide).
  • It acts on what it hears.
    • Emergency signs: it tells them to call 911 and alerts everyone at once.
    • Warning signs: it finds the nearest cooling or warming center (and knows if it's closed right now), offers a ride, books a callback, and phones the family within seconds.
  • It remembers. Supermemory lets the next call open with "Last time your AC was broken. Is it fixed?"
  • Escalation ladder. If the family member doesn't answer, Step Functions moves to the backup contact, then the caseworker.
  • Team dashboard. Caseworkers see who needs a human, what Kinwell is already handling, and who's safe. Gemini ranks who to call first, and every call has a case note, transcript, and "who's been told."
  • Crew Check for outdoor workers: shift check-ins, with supervisors alerted only on a red flag (worker privacy).

How we built it

  • Voice: Twilio Media Streams → our FastAPI server on ECS Fargate (behind CloudFront + ALB) → Amazon Nova 2 Sonic bidirectional streaming on Amazon Bedrock. The agent uses tools: flag_risk, find_cooling_center / find_warming_center, record_checkin, schedule_callback, end_call.
  • Safety rules in code, not just the prompt: a deterministic rules engine decides warning or emergency (for example, 90°F+ inside is a warning even with the AC "on"). The server refuses to hang up until the key answers are recorded and the cooling center is offered.
  • Escalation: AWS Step Functions ladder; EventBridge Scheduler + Lambda risk engine polling the NWS API nationwide.
  • Data: MongoDB Atlas: people, check-ins, alerts, and resources with a 2dsphere index and $geoNear for the nearest open center; caches for geocodes and forecasts.
  • Memory: Supermemory, one namespace per person.
  • AI: Gemini writes case notes, ranks the caseworker briefing, and runs the Family enrollment assistant (hardened against prompt injection, with server-side validation).
  • Resource Scout: a TinyFish web agent on Maritime reads city library and cooling-center pages for today's hours, so the agent never invents a place or a time.
  • Identity and location: Amazon Cognito for family sign-in; Amazon Location geocoding (Census fallback).
  • Frontends: React + Vite + Tailwind, hosted on AWS Amplify. The Family site is an installable PWA. Domain: heykinwell.us (.us is run by GoDaddy Registry).

Challenges we ran into

  • Latency. The first calls had about 8 seconds of silence. Asynchronous answering-machine detection, plus pre-warming the Nova Sonic session while the phone rings, brought the first word down to the network's own delay.
  • Models skip steps. The agent sometimes said goodbye before offering the cooling center, or forgot to hang up. We moved those guarantees into server-side guards and tool results instead of hoping the prompt holds.
  • Region constraints. Our AWS account only allows one infra region, while Nova Sonic runs in another, so we split the voice model and the infrastructure across regions.

Accomplishments that we're proud of

  • Real calls, real escalation: the agent hears "a little dizzy, about 95 inside," and the family's phone rings seconds later.
  • Consent-first enrollment and privacy by design: audio is never stored, and transcripts expire after 30 days.
  • Works anywhere the National Weather Service does, for heat and cold.

What we learned

Voice agents feel magical until they skip a step. Reliability comes from pairing the model's conversation with deterministic code for the parts that keep people safe.

What's next

  • Pilots with county heat-relief programs and senior centers
  • Rides through community transport partners
  • More languages
  • SMS fallback for people who can't take calls

Built With

  • amazon-bedrock
  • amazon-cloudfront
  • amazon-cognito
  • amazon-eventbridge
  • amazon-location-service
  • amazon-nova-sonic
  • aws-amplify
  • aws-ecs-fargate
  • aws-lambda
  • aws-step-functions
  • fastapi
  • gemini
  • mongodb-atlas
  • national-weather-service-api
  • python
  • react
  • supermemory
  • tailwindcss
  • tinyfish
  • twilio
  • vite
Share this project:

Updates

Submission history