SF Companion

Inspiration

Walking around San Francisco means juggling four browser tabs: Google Maps for directions, Citizen for crime, Twitter for what locals are saying right now, and Wikipedia for "wait, what is that building?" We wanted one companion that could plan a walk, narrate the neighborhood, and warn you about an incident that hit the news ten minutes ago — by voice, while your phone stays in your pocket.

What it does

SF Companion is a voice-first walking agent for San Francisco. You say where you want to go; it plans the walking route on a live map, fetches segment-by-segment weather, pulls real SFPD incidents along the path, scrapes fresh crime reporting from local outlets, and surfaces attractions worth a detour. The Claude-powered agent drives the map directly — when it mentions Coit Tower, the pin pulses; when it warns about a hotspot, the segment highlights. Text or voice, your choice.

How we built it

A federated GraphQL stack glued to a multi-tool Claude agent:

A TypeScript graphql-yoga BFF serving the full GraphQL contract with streaming subscriptions over an in-process bus. InsForge Postgres (PostGIS 3.6.1 enabled) for users and sessions; Redis for route and weather caching keyed by (lat, lng, hour). Mapbox Search Box and Directions handle geocoding and walking routes with an SF bbox check; Open-Meteo fills in per-segment weather. On top, an Anthropic Claude tool-use loop streams deltas, tool calls, and results through four wired tools — plan_route, get_route_context, get_user_location, show_on_map — each gated by a swappable authorize + logToolCall governance facade. Vapi voice, Nexla ingestion, TinyFish, Senso, Cosmo federation, and the frontend are scoped but not yet built.

Challenges we ran into

• Verifying PostGIS on InsForge — the architecture leaned on ST_DWithin over geography, so before we built anything we had to confirm the extension was actually available. It took a CLI deep-dive (db query "SELECT extname FROM pg_extension") and a tracked migration to lock it in as 3.6.1.
• Picking the right BFF primitive — InsForge ships two hosting options and only one of them (compute, Fly.io under the hood) supports the long-lived WebSockets and SSE that streaming subscriptions and Claude responses need. Working that out from docs marked "in progress" took longer than the decision deserved.
• Streaming a heterogeneous event union — the agent emits text deltas, tool calls, tool results, and a final ChatDone over a single GraphQL subscription. Wiring that to graphql-yoga meant writing a small async-iterator bridge over an in-process channel that filters by turnId and ends cleanly when the turn does.

Accomplishments that we're proud of

• Canonical GraphQL schema wired end-to-end on the dev server, with real resolvers for routing, weather, and the agent loop.
• Real Mapbox walking routes with SF bbox enforcement and per-segment splits.
• Open-Meteo per-segment weather, Redis-cached by (lat, lng, hour).
• PostGIS 3.6.1 enabled on InsForge, ready for along-route spatial queries.
• Streaming Claude agent with a four-tool loop (plan_route, get_route_context, get_user_location, show_on_map) over a single GraphQL subscription.

What we learned

• Federated GraphQL pays off the moment you have more than one subgraph that benefits from independent deploy cadence.
• "Long-lived" vs. "request/response" is the question to ask when picking a serverless host — most edge platforms quietly fail the first one.
• Pre-built vendor agents (TinyFish for scraping, Nexla for ingestion, Senso for RAG)

What's next for SF Companion

• Proper auth via InsForge OAuth (today we're anonymous-session only)
• Historical crime joining DataSF wg3w-h783 for >48h lookback and trend analysis
• Ambient voice narration — "tell me when I'm near something interesting" without prompting
• Safer-alternate-route suggestions when a segment crosses a recent incident cluster
• Generalize beyond SF: LA and NYC are the obvious next two cities given Socrata coverage
• Real Guild.ai integration for tool-call governance, replacing our file-based audit stub

Built With

• anthropic-claude
• insforge
• mapbox
• nexla
• postgis
• senso.ai
• tinyfish
• typescript
• vapi
• wundergraph-cosmo