RefugeeReach

RefugeeReach is an AI-powered crisis companion for displaced people, designed to provide real-time voice interaction, aid search, emergency escalation, and news collection.

Comment

RefugeeReach - AI Emergency Companion for Displaced People

Inspiration

The idea for RefugeeReach came from watching the ongoing displacement crises around the world - Gaza, Sudan, Ukraine - and realizing that millions of people fleeing conflict face an overwhelming information gap at the worst possible moment. They don't speak the local language, they can't read official documents, and they have no idea where the nearest hospital or shelter is. Existing tools mostly stop at translation. We wanted to build something that goes further: a trusted digital companion that can actually act on behalf of someone in crisis.

We drew direct inspiration from projects like World Monitor for the news intelligence feed, and from the multi-agent, multi-modal patterns that tend to win hackathons - not because we wanted to win, but because those patterns genuinely fit the problem. A displaced person's needs are inherently multi-modal: they might speak a question, photograph a document, or just press an SOS button. One model can't do all of that well.

What It Does

RefugeeReach is a voice-first, multilingual PWA that combines several capabilities into a single interface:

  • A real-time voice assistant powered by Amazon Nova Sonic that lets users speak and get immediate, actionable guidance
  • An SOS button that broadcasts GPS coordinates to emergency contacts via AWS SNS
  • An aid resource finder that queries OpenStreetMap and UNHCR data to locate nearby hospitals, shelters, water points, and refugee camps
  • A crisis detection engine that analyzes user messages (via keyword matching or LLM) and auto-escalates emergencies
  • A news intelligence pipeline that ingests articles from trusted sources, deduplicates them using semantic similarity, clusters them into events, and delivers severity-scored alerts

How We Built It

The system is structured as a multi-agent architecture using the Strands Agents framework on top of Amazon Nova models.

The orchestrator agent (Nova Lite) classifies user intent and routes to specialized sub-agents:

Agent Model Role
Orchestrator Nova Lite Intent detection, routing
Voice Agent Nova Sonic Real-time speech-to-speech via WebSocket
SOS Agent Nova Lite Crisis analysis, SOS alert triggering
Aid Locator Agent Nova Lite Nearby resource search
News Agent Nova Lite Geopolitical event summaries

The backend is a FastAPI server exposing both REST endpoints (/chat, /sos, /aid/nearby, /news) and a WebSocket endpoint (/voice/stream) for bidirectional audio streaming. The voice pipeline works like this:

  1. Browser captures microphone audio at 16 kHz mono PCM
  2. Audio chunks are base64-encoded and sent over WebSocket
  3. The server bridges them to Nova Sonic's real-time API
  4. Nova Sonic returns audio deltas + transcripts + tool calls
  5. Tool calls (e.g., find_nearby_aid, detect_emergency) execute server-side and feed results back into the conversation

The aid locator uses the Haversine formula for distance ranking:

$$d = 2r \arcsin\left(\sqrt{\sin^2\left(\frac{\Delta\phi}{2}\right) + \cos(\phi_1)\cos(\phi_2)\sin^2\left(\frac{\Delta\lambda}{2}\right)}\right)$$

where $r = 6371\text{ km}$ is Earth's radius, and $\phi$, $\lambda$ are latitude and longitude in radians.

The news pipeline is a full event-driven system with RabbitMQ workers:

Ingest -> Normalize -> Deduplicate -> Analyze (LLM) -> Policy -> Deliver (WhatsApp)

Each stage is a separate worker consuming from a queue. Deduplication uses cosine similarity on sentence embeddings to cluster articles into events. The analysis worker calls Nova Lite to generate severity scores ($0.0$–$1.0$), topic classification, and summaries. All data lands in PostgreSQL with Alembic migrations.

The frontend is a React + Vite PWA with service worker registration for offline support, MapLibre GL for rendering aid locations, and a custom WebSocket voice client with a jitter buffer for smooth audio playback.

Challenges We Ran Into

The voice feature on iOS was the biggest headache. React's StrictMode unmounts and remounts components in development, which wiped our useRef guards and caused duplicate WebSocket connections. On top of that, iOS Safari fires both touchend and a synthetic click on a single tap, so the mic button was getting hit twice. The fix involved moving the session state to a module-level singleton that survives React lifecycle and adding a 400 ms timestamp debounce on the toggle handler.

Nova Sonic's WebSocket protocol required careful handling - the handshake sequence (session.created -> session.update -> session.updated) has to happen in exact order, and tool call results need to be sent back as function_call_output items with the correct call_id. Getting the audio encoding right (PCM16 <-> Float32 <-> base64) across the browser and Python took several rounds of debugging.

The news pipeline's deduplication was tricky to tune. Too aggressive and distinct events get merged; too loose and the same story appears five times. We settled on a cosine similarity threshold that balances precision and recall, but it's still not perfect for breaking news where early reports are vague.

Cost management was a constant consideration. We kept Nova Lite as the workhorse for all text reasoning (at $\$0.06$ per million input tokens) and only invoke Nova Sonic for active voice sessions and Nova Pro for document uploads. Lambda + DynamoDB scale to zero between demos, keeping the total hackathon spend under $\$15$.

What We Learned

  • Multi-agent orchestration with tool use is powerful but requires disciplined prompt engineering - the orchestrator needs very clear routing instructions or it tries to answer everything itself
  • Real-time audio streaming in the browser is a minefield of platform-specific quirks (iOS audio context restrictions, sample rate mismatches, ScriptProcessorNode deprecation)
  • Building for low-literacy, high-stress users forces you to think differently about UX - every extra tap or confusing label is a real barrier
  • Event-driven architectures with message queues (RabbitMQ) make it much easier to reason about and debug each pipeline stage independently

Built With

Amazon Nova Sonic, Amazon Nova Lite, Amazon Nova Pro, Amazon Bedrock, AWS SNS, AWS Lambda, Python, TypeScript, Strands Agents, FastAPI, Uvicorn, React, Vite, React Router, React Leaflet, Leaflet, React Markdown, WebSockets, PostgreSQL, SQLAlchemy, Alembic, psycopg2, RabbitMQ, Pika, Redis, Docker, Nginx, OpenStreetMap Overpass API, UNHCR API, OpenRouteService API, NewsAPI, boto3, httpx, feedparser, BeautifulSoup4, Trafilatura, pgvector, RapidFuzz, OpenAI API, Pydantic, python-dotenv, geocoder, PyAudio, Service Workers, PWA

Built With

Share this project:

Updates