RelayGoal

Can't use the phone? Type the goal. A CALL-E agent makes the whole call and hands back a verified answer bound to the exact words that were said - not a summary to trust.

Comment

Inspiration

Roughly 60% of Deaf and hard-of-hearing Americans say the phone is still a barrier - refills, appointments, benefits offices, all gated behind a call they can't make. Every relay today (TRS/VRS, captioned telephony) is synchronous: you must sit on the live call reading captions, because federal relay funding requires it. We wanted the errand done without the user on the line - and, because they never heard it, a way to prove what was actually said.

What it does

You type a goal in plain language ("Call the pharmacy, check if my metformin refill is ready, and ask when they close"). A CALL-E agent places and conducts the entire outbound call, discloses that it is an assistant calling on someone's behalf, works through phone menus, and captions the conversation live. When it hangs up, you get a verified-result card: each answer is either Verified — bound to the exact transcript sentence the callee spoke — or Not confirmed, with "call them yourself" and the number. A compare mode calls several places at once and recommends one only when both its gating fact and its comparison value were verified on the call. On a phone the live call takes over the whole screen with large captions, and the app installs to the home screen.

How we built it

  • CALL-E goal-driven tasks with structured extraction: result_schema, completion_confidence, evidence, and transcript_turns, behind one CalleLike interface with a credential-free FakeCalle dry run.
  • A reusable verified-phone-outcome core: an answer is verified only when the call completed, confidence clears a threshold, and the value is grounded in a transcript turn the callee (never the agent) actually spoke. Everything else fails closed. No confidence percentage is ever shown to the user.
  • Next.js + TypeScript; an installable PWA (web app manifest + service worker); accessible by design (Geist body with a Fraunces display face, full keyboard/screen-reader support, colourblind-safe verification states carried by icon + shape + text, high-contrast / text-size / reduced-motion controls).

Architecture

Data flow (a rendered diagram is in the project gallery): a user types a goal -> the Next.js app (the browser /api/call and /api/batch routes are fake-only and never dial) -> the verified-phone-outcome core, which picks a client: FakeCalle stub by default (canned 555 scenarios, no dial), or the @call-e/calle SDK only on an explicit opt-in, to an authorized recipient over an HTTPS-allowlisted host -> runVerifiedGoal validates E.164 and places exactly one call -> CALL-E dials, discloses, converses and captions live, returning status, structuredResult, completionConfidence and transcriptTurns -> verifyOutcome binds each answer to the callee's verbatim transcript turn: grounded in a quote becomes Verified (shown with that quote), otherwise Not confirmed ("call them yourself", fail-closed); completion_confidence is only a gate and is never shown -> the outcome is phone-masked into a proof-of-call receipt (each answer bound to its quote, a captioned transcript, and full-screen live captions on mobile). An offline eval harness (9 fixtures + 54 adversarial mutations) guards verifyOutcome at 0 false verifications.

Open-source contributions (found while building on CALL-E)

  • Submission app: awesome-phone-call-agents#599 — RelayGoal under apps/typescript/relaygoal/.
  • calle-docs#56: the Goal Run create (201) example showed in_progress for a not-yet-dialed run; fixed to queued to match the OpenAPI example and the GoalRunStatus enum.
  • server-sdk-typescript#25: the Quickstart README used a calle_test_key placeholder that a real key's iams_live_ prefix contradicts (a wrong-prefix key 401s identically to no key); aligned it with the Authentication guide. Found while wiring our live path.

Challenges we ran into

  • Verifying a call you can't hear. Symptom: an agent will confidently summarize a call that never actually confirmed the fact. Diagnosis: the summary and the transcript can disagree. Decision: bind every answer to a callee transcript turn and refuse to answer otherwise. Verified result: a 54-mutation adversarial harness holds at 0 false verifications.
  • The honest limit. Two facts in one spoken sentence can't be separated by quote-grounding — a within-sentence value swap is caught only 3 of 17 times. Rather than hide it, we show the quote for the user to check and never a confidence score.
  • Live-path safety. A key in the environment must never dial on its own, so live calls are gated behind a double opt-in (RELAYGOAL_LIVE=1 and CALLE_API_KEY), with strict E.164 and payload-bound idempotency.

Accomplishments that we're proud of

  • Real live calls through the CALL-E SDK to the official US test hotline: 101 seconds, 2 of 2 answers verified against verbatim quotes, disclosure found at 0:03.
  • Installable, phone-native accessibility: the live call as a full-screen captioned experience.

What we learned

  • "An AI made a call" is worth little to someone who couldn't hear it; "here's the answer, and here's exactly what they said" is worth a lot. The verification layer is the product, not a feature.
  • Honest limits build more trust than a confidence number: showing the quote and saying "not confirmed" beats showing a 78%.

What's next for RelayGoal

  • A recording-consent preamble and per-state consent handling on the live path.
  • Push-notified call completion on the installed PWA.
  • More goal categories beyond low-harm errands, held to the same fail-closed bar.

Built With

Share this project:

Updates

Submission history