PitchOps

Inspiration

Live stadium events run on controlled chaos. During a FIFA watch party halftime, a projector goes dark, a crowd surges, a smoke alarm triggers — and the on-call venue engineer is standing in a corridor with tools in hand, no free hand to type, and seconds to act.

We wanted to build the operations copilot that engineer deserves: something that knows the entire venue — every breaker, every rack, every playbook — and can answer a spoken question with a spoken, safety-checked answer in under five seconds.

The constraint that inspired everything: no typing allowed

What it does

AngelMC is a voice-first mission control dashboard for live event venue engineers.

The engineer holds a button, asks a question out loud, and hears a concise, safety-verified answer back:

"Where is the breaker for Projector B?" "Go to the AV Booth first and check AC-2 before touching the electrical panel."

Behind that exchange is a full operations loop:

Voice input via Wispr Flow speech-to-text, with browser SpeechRecognition as fallback Mission control agent powered by Claude, reasoning over a loaded venue context folder — assets, floor map, safety playbooks, incident playbooks Voice output via ElevenLabs, with browser SpeechSynthesis as fallback A live incident feed that logs queries and simulation events in real time An interactive blueprint map that highlights the relevant venue zones from every agent response Three incident simulation buttons — projector failure, network rack alert, smoke detector — each triggering realistic log sequences and map highlights A safety guardrail layer that intercepts any query containing smoke, fire, alarm, or electrical hazard keywords and forces escalation language before the response reaches the engineer One-tap incident report generation with a copy button, ready to paste into a ticket or a Devpost demo note

How we built it

Microphone → Wispr Flow STT → Claude (mission control agent) → ElevenLabs TTS → Engineer hears the answer Frontend: React, TypeScript, Vite, Tailwind CSS. Three-column dark operations dashboard — blueprint map, incident feed, voice assistant + report panel. The useVoiceInput hook wraps MediaRecorder for audio capture. The useAudioPlayback hook handles blob-to-audio playback. All API keys stay off the client.

Backend: Node.js, Express, TypeScript. Eight REST endpoints. A context loader reads five Markdown files from server/context/ at session start and caches them per session — the agent always has full venue knowledge. Safety guardrails run as a post-processing pass on every agent response before it reaches the wire.

Agent: Claude via the Anthropic SDK. The system prompt instructs strict JSON output, safety-first language, and zero invented venue facts. The response schema drives the UI directly — highlightedLocations updates the map, suggestedSpokenInstruction goes to ElevenLabs, nextSafeChecks populates the response card.

Context: Five Markdown files define the entire venue — assets with wiring paths, a floor map with all eleven locations, safety rules, and incident playbooks for every scenario demoed.

Challenges we ran into

Getting Claude to return clean JSON every time. Early prompts produced markdown-wrapped responses that broke parsing. The fix was a stricter system prompt, a code-fence strip pass before JSON.parse, and a typed fallback response for any parse failure so the engineer always hears something safe rather than silence.

Safety guardrails that can't be bypassed by the model. We couldn't trust the model alone to refuse unsafe electrical instructions under every rephrasing. The solution was a deterministic post-processing layer in safetyGuardrails.ts that scans the original transcript for trigger keywords and overwrites the response with escalation language regardless of what the model returned.

Voice UX on a hold-to-speak button. mousedown/mouseup and touchstart/touchend need to be wired symmetrically, and the recording state has to block further presses until the full agent + TTS loop completes. Getting the status machine (idle → recording → transcribing → thinking → speaking → idle) right without race conditions took several iterations.

Keeping secrets off the frontend. Every API call that touches Wispr Flow, ElevenLabs, or Anthropic routes through the Express backend. The frontend only knows VITE_API_BASE_URL. The backend .env is gitignored and never committed.

Accomplishments that we're proud of

The full voice loop — speak a question, hear a grounded answer with the correct breaker number — works end-to-end in a single hold-to-speak gesture Safety guardrails are deterministic and cannot be talked around: smoke + alarm + fire + electrical hazard queries always escalate, regardless of model output The blueprint map lights up the exact locations the agent identifies, turning an abstract text answer into a spatial pointer the engineer can act on immediately The agent never invents venue facts — it says what it knows and flags what needs confirmation The entire demo runs without a single typed character

What we learned

Voice-first UX requires a tighter state machine than click-first UX — there is no cursor, no visible affordance, and no second chance before the engineer acts on what they hear Deterministic safety layers matter more than model instructions alone — the model is probabilistic, the guardrail is not Structuring the agent output as typed JSON that directly drives UI state (map highlights, spoken text, check list) removes an entire class of rendering bugs Loading venue knowledge from flat Markdown files at session start is faster to iterate on than fine-tuning or RAG for a hackathon scope — the context fits in a single prompt

What's next for PitchOps

Real-time asset state: WebSocket feed from actual AV controllers and network switches so the map reflects live equipment status, not just simulated events Multi-engineer sessions: Shared incident feed so two engineers working different zones see each other's queries and the agent's responses Offline fallback mode: Cache the context and a lightweight model locally so the assistant works when venue Wi-Fi goes down — exactly when it is needed most Post-event report export: Auto-generated shift summary in PDF format, ready for the venue operations log

Built With

• anthropic
• css
• elevenlabs
• express.js
• html
• node.js
• react
• tailwind
• typescript
• vite