What it does
Relay turns a responder's spoken report into shared mission context and an accountable next action.
A responder speaks through a Vexo bracelet. Relay saves the incoming audio before AI processing, and Gemini transcribes the report, extracts the important details, and proposes actions using the mission's accumulated context. The shared dashboard keeps reports, hazards, unresolved questions, and action history together. A dispatcher can trace a service request back to the report that prompted it, review it, and see the resulting acknowledgment.
Our working demo follows a physical wristband report through a Gemini-selected logistics request, a receipt saved by a controlled test service, and a second Gemini agent's confirmed follow-up. Each action stays linked to its source report and the receipt it depends on.
Inspiration
During a changing incident, each responder sees only part of the situation. A spoken update can contain a new hazard, a correction, or an unmet need. The team still has to connect that information to what everyone else knows and decide what to do next.
We built Relay so a responder can contribute by speaking while the dispatcher gets a shared record of what was reported, what remains uncertain, and which requests have been acknowledged.
What makes Relay different
Relay carries context from the original audio through the proposed action and destination receipt. That lets the team check why an action was suggested and whether the destination actually acknowledged it. A follow-up can use a confirmed result rather than assume the previous step succeeded.
Relay also includes an MCP server for compatible AI agents such as Claude Code, Codex, Cursor, and Gemini CLI. They can read a mission, follow changes, and propose actions that cite real report IDs. The dashboard labels external-agent proposals and holds them for dispatcher approval. The MCP server uses the same server validation and access controls as the dashboard; it does not expose mission reset or band controls.
How we built it
A Python Bluetooth bridge connects the Vexo band to a TypeScript/Node server and React dashboard. Gemini handles transcription, structured report extraction, mission planning, and optional spoken briefings. Server-sent events keep mission views synchronized.
We save incoming audio before model processing and run the AI work in background jobs, so a slow model call does not block new reports. Stable event IDs prevent duplicate intake on retry. Persisted jobs and receipts support recovery after restart, and report ordering preserves newer corrections when older processing finishes late.
The controlled execution workflow verifies a destination's saved receipt before asking a separate Gemini agent to plan a dependent follow-up. The MCP server is a thin client over Relay's HTTP API, keeping validation and stale-version checks in one place.
Challenges we ran into
New speech arriving during planning exposed our hardest bug: competing planners repeatedly invalidated one another until a saved report was marked failed. We reproduced the failure in tests, serialized planning per mission, and made Relay re-evaluate against new context automatically. Intake stays available while planning catches up. Regression tests failed before the fix and passed afterward.
Accomplishments that we're proud of
Two consecutive physical wristband rehearsals completed the full request-to-follow-up workflow in 40.9 and 39.3 seconds, with zero manual retries. Timing began when the band was listening and included speech, Gemini processing, the controlled service's receipt, and the confirmed follow-up. Bluetooth pairing happened beforehand. Both runs preserved the spoken quantities and locations.
The combined release passed 83 Node tests and a production build. The physical rehearsals ran on the fixed candidate before the publication merge; the release verification records that distinction.
Physical demo evidence ยท Release verification
What we learned
Saving the original report, preserving attribution, and handling retries matter as much as transcription quality. A useful shared mission view lets people see uncertainty and verify outcomes. Outside AI agents need the same evidence and approval rules as the rest of the system.
What's next for Relay
Our next milestones are longer capture and recovery tests, simultaneous use of multiple bands, stronger participant permissions, and a verified hosted deployment. Real-service integrations would follow controlled testing.
Relay is a local, single-band hackathon prototype. The demonstrated logistics workflow uses a controlled test service; emergency-service dispatch is simulated. Multi-hour physical endurance and a hosted deployment remain unverified. Development and documentation were AI-assisted.
Built With
- bluetooth
- express.js
- google-gemini
- node.js
- python
- react
- typescript
- vexo
