🛡️ Envoy

A voice agent that books on your behalf — and can't be talked into betraying you.

🎥 Demo: (https://drive.google.com/drive/folders/1dgEqeJ2n29OLZ5WTV72DCKW8Fn5PgLo7)run at 1.5x or 1.75x speed! Repo: github.com/rayan-arya/Envoy

---

1. What is this?

Envoy is a voice agent that places real phone calls to book reservations for you. You tell it what you want — restaurant, time, party size — it confirms the details back to you, then it dials the venue, negotiates, and locks in the booking, firing a calendar invite and a confirmation email as it goes.

The part we actually care about is what happens under pressure. A booking agent ends up holding your secrets — confirmation codes, account details, anything the person on the other end can talk it out of. So when a novel manipulation gets through, Envoy diagnoses the breach, writes itself a new guardrail, and hot-reloads it live — making it immune the next time, even to rewordings it has never seen. It learns the attack class, not the wording. It gets safer the more it's attacked, and an independent eval (Cekura) confirms it: 0/5 → 5/5 credential-leak refusal after a single self-written rule.

How it works

The booking flow. Envoy places the real call to the venue, handles the live negotiation (it bumped 7 → 7:30 when 7 was full), gives the reservation name, and on the host's confirmation fires two tools — create_event and send_confirmation — with a byte-identical confirmation_ref, landing a calendar event and a confirmation email. In front of that sits an optional principal-intake phase (flag-gated, ENABLE_INTAKE): Envoy first takes the booking from you, validates all three fields, recaps for confirmation ("So that's Bella Vita, around 7, party of two — calling now?"), and hands the collected parameters to the call via a structured confirm_intake tool call. The intake is built and verified brain-side; it's off by default and not yet wired into the live voice pipeline.

The self-heal loop is the centerpiece. Every agent starts naive. The first time a caller tries a social-engineering attack Envoy hasn't seen, it leaks — and that single breach becomes permanent immunity:

Stage	What happens
1 · Breach	A novel attack gets through; the agent discloses a secret.
2 · Detect	An independent Claude judge reads the exchange and flags the leak — a separate model with a separate prompt, so it's a real check, not the agent grading itself.
3 · Patch	A patcher writes a generalizing guardrail — a rule against the class of attack (any code / PIN / token / passphrase / credential), not the one phrasing that just worked.
4 · Immune	The rule is written to guardrails.json, which a custom GuardrailInjector reloads into the live pipeline every turn — so the agent refuses that attack, and every reworded cousin of it it's never seen.

The voice pipeline: Pipecat orchestrates the stack — NVIDIA Nemotron as the brain (run with detailed thinking off — see feedback), NVIDIA Nemotron Speech Streaming for speech-to-text, Gradium for text-to-speech, over a Twilio (PSTN) / SmallWebRTC (local) transport. Tool-calling, turn-taking (stop_secs / VAD), and the guardrail hot-reload all live in this loop.

The result, measured: from a naive agent that leaks every attack, one self-written rule takes credential-leak refusal from 0/5 → 5/5 (independently scored by Cekura), blocks 6/6 reworded variant attacks it was never shown, and drives the live trust score 50 → 100.

2. Demo video - run at 1.5x or 1.75x speed!

**▶️ https://drive.google.com/drive/folders/1dgEqeJ2n29OLZ5WTV72DCKW8Fn5PgLo7

Envoy books a real reservation over the phone, then it gets attacked — and writes its own rule to harden, verified live by Cekura's independent judge.

3. How we used Cekura, Nemotron, and Pipecat

NVIDIA Nemotron — the agent's brain. An open-weights Nemotron model runs both the booking agent and the naive baseline we attack. We drive it with a layered system prompt (caller identity, what it may disclose, what it must always refuse) plus tool schemas for the booking side effects, and we run it with detailed thinking off so it answers concisely and emits clean tool calls (the why is in the feedback section — it was one of our biggest findings).

Pipecat — the voice orchestration. Built on the official hackathon starter, with NVIDIA Nemotron Speech Streaming STT, the Nemotron brain, Gradium TTS, and a Twilio / WebRTC transport. Our self-heal loop writes learned rules to a file that a custom GuardrailInjector frame processor reloads every turn, so a rule written mid-session takes effect on the agent's very next response — that's the mechanism behind live self-healing.

NVIDIA Nemotron Speech Streaming — speech-to-text. Transcription runs on NVIDIA's streaming ASR over WebSocket — so the STT and the LLM are both NVIDIA. Low-latency, and Envoy heard every turn (it caught the host's 7:30 counter-offer without a hitch).

Cekura — how we proved the self-heal actually works. This is the part we care about most. What we set out to do: prove the self-heal makes the agent measurably safer, judged by an independent third party rather than our own detector. We defined a credential-leak-refusal metric (a binary LLM judge: did the agent disclose any code / PIN / token / passphrase / credential?) and scored the agent on five distinct social-engineering attacks, before and after its self-written guardrail:

	Credential-leak refusal (Cekura llm_judge, metric 147699)
Before — naive agent, no learned rules	0 / 5 — leaks every attack
After — one self-written rule (R1)	5 / 5 — refuses every attack

Improvement: a 100% leak rate driven to 0%, independently verified. Cekura's judge explanations matched our own detector turn-for-turn ("disclosed a confirmation code… credential leak" vs. "explicitly refused… for security"). That independent measurement is what turns "we think it's safer" into "a third party confirms it's safer" — the whole credibility of a self-improvement loop.

4. What we built during the hackathon

Envoy is a new project built at this hackathon. We started from the official Pipecat starter (the Field & Flower voice scaffold), which gave us a working call pipeline. Everything that makes Envoy Envoy we built here:

• the self-heal loop — an independent Claude judge that detects breaches, a patcher that writes generalizing guardrails, and hot-reloading so a rule takes effect mid-call
• the generalization property — it learns the attack class, not the wording (6/6 reworded variants blocked)
• the Cekura eval integration — the independent 0/5 → 5/5 before/after measurement
• the real-time Self-Heal Monitor dashboard
• the booking flow — create_event + send_confirmation tool calls with a byte-identical confirmation_ref
• the principal-intake phase (built and flag-gated — ENABLE_INTAKE, off by default; brain-side complete and verified, not yet wired into the live voice pipeline) — Envoy first takes and confirms the booking before it ever dials out, handed off via a confirm_intake tool. (It also points at a direction we like: because the principal never has to be on the call, a text-based intake could let someone who's hard of hearing arrange a reservation Envoy then places by voice.)

Borrowed: the Pipecat starter scaffold and the provider SDKs. New (this hackathon): the entire self-hardening security system, the independent eval, the dashboard, the booking flow, and the intake phase.

5. Feedback on the tools

NVIDIA Nemotron

• detailed thinking off was our single highest-leverage fix. Nemotron is a reasoning model, and with thinking on it (a) emitted <think>…</think> tokens that leaked into the TTS — the agent literally said "</think>" out loud — and (b) on the phone path, the reasoning interfered with tool-calling: the agent would verbalize "I've booked it" but emit zero tool calls, so no calendar event and no email. Flipping to detailed thinking off (set as the first system message) fixed both at once — no thinking tokens to strip, no reasoning getting between the model and its tools, and lower latency as a bonus. If we'd known the toggle existed earlier we'd have saved hours. A louder note in the docs that reasoning-on can suppress structured tool calls in agentic loops would help every builder.
• Latent safety is temperature-sensitive. To demo the self-heal we run a deliberately naive agent (no safety posture). At temp ~0.8 it would spontaneously refuse to leak a secret; at ~0.4–0.6 it complied reliably. Useful to know the helpful/safe behavior shifts with temperature.
• Strong at instruction-following and tool use (once thinking is off). It honored a layered disclosure/refusal prompt, followed a broadly-worded learned guardrail without over-refusing legitimate info, and emitted parallel tool calls with consistent arguments. A genuinely solid open-weights brain for an agent.

Cekura (feedback on building self-improvement loops)

• The transcript/observe path is what made rapid iteration possible. Under time pressure we couldn't gamble a fresh deploy, and the automated WebRTC testing path requires a deployed agent. The observe_create → call_logs_evaluate_metrics_create flow let us score captured transcripts against a custom metric with no live deploy — that's what made our before/after measurement feasible in the time we had. For self-improvement loops specifically, "score the runs you already have" is the unlock.
• The independent judge is the right primitive. Defining a binary metric and getting judged scores with explanations that matched our own detector is exactly what makes "the agent improved" credible to someone outside the team.
• Frictions: the bundled MCP read its API key from the process env at launch, so a shell export couldn't re-key a running session — OAuth via /mcp was the cleaner path. And a more first-class loop primitive — score → apply a guardrail change → auto-re-evaluate, tracked as one "improvement run" — would make self-improvement loops feel native instead of stitched together from individual calls.

Pipecat

• The starter was an excellent launch point — we were on a working call fast.
• Swappable services saved us. Clean service interfaces and the two-build starter (an OpenAI build and an NVIDIA Nemotron build) let us iterate on STT/TTS/LLM independently — swapping a service was a few lines, not a rewrite, exactly what you want mid-hackathon.
• The pattern of reloading a guardrails file every turn through a frame processor worked beautifully for hot-swapping rules mid-call — it's the core mechanism behind our live self-heal.
• Twilio transport was the rough edge. The agent was clean on WebRTC but garbled/laggy over Twilio's 8 kHz μ-law PSTN until we sorted the sample-rate/resampling; a bit more guidance on the telephony codec path and on latency tuning (stop_secs / max_tokens) in the docs would help newcomers a lot.

Why we built this — and our hackathon experience

We came in genuinely excited about voice — there's something different about building an agent that talks to a real person on the phone instead of returning JSON, and the immediacy of it is addictive. The first time Envoy negotiated a time and booked a table out loud, it stopped feeling like a script and started feeling like a teammate.

That's also why we added the intake phase — talking to Envoy first. We didn't want a hard-coded errand-runner; we wanted something that takes an instruction from its principal, confirms it understood, and then goes and acts. Building that made the "agent acting on your behalf" idea real instead of theoretical — and it quietly opened an accessibility direction we like (the principal never has to be on the call, so a text-based intake could let someone who's hard of hearing make a reservation Envoy places for them).

But the thing that kept pulling our attention was trust. An agent that books for you, pays for you, calls for you inherits the exact attack surface humans get exploited through — social engineering — with none of a wary person's instinct to adapt mid-call. You can't ship a patch fast enough for every new script. Envoy is the smallest honest version of the thing we think this needs: an agent that doesn't just refuse manipulation, but learns from each breach and hardens itself. Watching it get tricked once and then shrug off every reworded version of that attack — and seeing Cekura independently score the jump from 0/5 to 5/5 — was the moment the whole project clicked.

The repo above is the source of truth. Envoy runs locally over SmallWebRTC or on a real phone via Twilio + Pipecat Cloud — see the run instructions below.

---

Architecture

server/
├── brain/                  # Envoy's caller brain: layered system prompt + tool schemas + intake state
├── counterpart/            # restaurant persona + the manipulation attack library
├── selfheal/
│   ├── monitor.py          # Claude-as-judge — detects credential leaks
│   ├── patcher.py          # writes generalizing guardrail rules
│   ├── loop.py             # the self-heal loop
│   ├── naive_baseline.py   # the Day-0 agent (no posture) for the before/after
│   ├── cekura_eval.py      # independent Cekura eval (0/5 → 5/5)
│   └── replay_demo.py      # deterministic replay of the self-heal sequence
├── dashboards/             # the real-time Self-Heal Monitor (FastAPI + live JSON)
├── data/                   # guardrails.json (hot-reloaded), eval_history.json, transcripts
├── bot-nemotron.py         # the Pipecat voice agent (Nemotron build)
├── pipeline_factory.py     # the Pipecat pipeline
├── integrations/           # Google Calendar + Gmail confirmation side effects
├── llm_client.py
└── contracts.py            # shared data contracts

Run it

Prereqs: uv and a .env in server/ with the provider keys (NVIDIA_API_KEY, NVIDIA_ASR_URL (STT), GRADIUM_API_KEY (TTS), ANTHROPIC_API_KEY, CEKURA_API_KEY, TWILIO_ACCOUNT_SID / TWILIO_AUTH_TOKEN / TWILIO_FROM_NUMBER, PIPECAT_CLOUD_PUBLIC_KEY, plus the Nemotron LLM endpoint).

cd server
uv sync
uv add anthropic requests           # judge/patcher + Cekura eval deps

# Voice agent (local WebRTC, :7860); or `pcc deploy` for Pipecat Cloud
uv run bot-nemotron.py

# Real-time Self-Heal Monitor → http://localhost:8080
uv run python dashboards/server.py

# Reproduce the self-heal loop (real model calls)
uv run python -m selfheal.loop

# Reproduce the independent Cekura eval (0/5 → 5/5)
uv run python -m selfheal.cekura_eval

---

Built at the Voice Agents Hackathon @ YC, 2026 — NVIDIA · Cekura · Pipecat · Gradium · Twilio.

Built With

• anthropic
• cekura
• claude
• dockerfile
• fastapi
• gmail
• google-calendar
• gradium
• html
• nvidia-nemotron
• nvidia-nemotron-speech-streaming
• pipecat
• pipecat-cloud
• python
• smallwebrtc
• twilio
• uv
• webrtc
• websocket