SF-BEACON (State Farm Basic Emergency Assistance & Cognitive Outreach Network)

Inspiration When a tornado or flash flood hits, the people who need help most are often the ones who can't ask for it. 911 lines jam, cell service gets spotty, and a lot of State Farm customers don't hear from their insurer until they file a claim days later. We wanted State Farm to be the one reaching out first, in the first few minutes, to every policyholder in the storm's path, and to do it without leaving anyone overlooked.

What it does

BEACON checks on policyholders during a disaster and gets help to the ones who need it most.

  • Set up before the storm. Policyholders add their home address, household details, and a safe room for each hazard. The app suggests one based on the home. There's no background location tracking. We work from the registered address.
  • Alerts come from the map, not zip codes. We pull live National Weather Service alerts and use PostGIS to find every policyholder whose home is inside the warning polygon. Each person gets a push notification and a 30-second check-in screen: I'm safe, I'm in my safe room, or I need help.
  • If you don't answer, we call. When the timer runs out, a voice agent calls the policyholder through Photon. It always opens with three fixed safety questions, then follows up based on the person's profile and what they say.
  • Nemotron is the brain of the call. It writes each person's follow-up questions, replies live, and extracts a structured triage result: status, urgency, emergency type, and needs. Behind it is a multi-step agent with tool calling. A coordinator decides who to call next, a triage agent scores each call, and an escalation agent decides what happens next.
  • Real emergencies go to 911, and nothing else does. Life-threatening situations create a 911 escalation with the caller's address, safe room, and the phone's precise GPS position. That position is only shared at that moment. A human responder reviews it before anything is "sent". Needs like medication or transport go to a State Farm responder queue instead. People we can't reach after several tries get a welfare check.
  • A responder dashboard shows a live map of everyone in the alert area, a crowd-sourced risk grid by neighborhood, the 911 and assistance queues, and a running feed of what the agent is doing and why.

How we built it

  • App: Expo / React Native, tested on iPhone through Expo Go. It registers each user with the backend and checks for alerts, and it sends check-ins back.
  • Backend: Node, TypeScript, and Express, on Postgres with PostGIS. A separate dashboard in React and Vite has a control panel where we send a test alert to any area and watch it reach the phone.
  • Voice: a Photon number connects through SIP to LiveKit, where our agent runs the call. Deepgram handles speech-to-text and Cartesia handles text-to-speech.
  • Nemotron: Nemotron 3.5 Lightning with thinking turned off, running about half a second per turn. We ran it on our own Brev GPU, with NVIDIA's hosted API as an automatic fallback. We also trained a small PyTorch Lightning risk classifier on Nemotron embeddings, used as an extra opinion during live calls.
  • Demo data: a simulated Houston flash flood with 1,500 policyholders inside the warning area and 100 outside.

Challenges we ran into

  • Safety can't depend on the model. The core questions, the 911 instruction, and the rule that the agent never promises help is coming are all enforced in code. If Nemotron is slow or returns junk, the call continues from a fixed script.
  • Too many false 911s. Our first version hung up and told people to call 911 the moment they said the word "emergency", including when they were asking about our emergency. It also treated "nobody is trapped" as danger. We rebuilt detection so 911 only triggers on a direct ask or a specific life threat, and we made it aware of negation.
  • Messy calls. Voicemail, silence, "stop calling me", wrong numbers, and people who don't speak English each needed their own behavior. We never leave a voicemail. An answered call with no speech counts as unresponsive.
  • The pipeline itself. We fought Photon SIP rejections on some dials, voice-agent start-up timeouts, and NVIDIA rate limits. Getting a phone and a laptop to talk on school Wi-Fi meant tunnelling both Expo and our API.

Accomplishments we're proud of

  • The whole loop works end to end: alert, push, check-in, phone call, triage, and a 911 payload on the dashboard.
  • Our danger judgment scored 100% on a 23-line evaluation set we wrote for it.
  • The app gets alerts from home addresses and mapped safe rooms instead of constant GPS tracking.
  • Calls feel human. The agent varies its acknowledgements, answers a caller's questions with general safety guidance, and stays calm.

What we learned

  • How emergency dispatch data is structured for first responders. Our 911 payloads are modeled on RapidSOS-style telemetry. They are simulated, and a human confirms each one.
  • How to build voice agents where code, not the model, owns the safety-critical decisions.
  • How to use a low-latency LLM for structured extraction mid-conversation.

What's next

  • Wire the escalation payloads to a real dispatch integration.
  • Turn texting back on as a fallback. The code is there, but we ran calls-only for the demo.
  • Bluetooth mesh relay so nearby phones can pass check-ins along when towers are down.
  • Smart home sensors (smoke, water, locks) feeding into the same triage.

Built With

Share this project:

Updates

Submission history