Steve Assist

Inspiration

Call a dentist's office at 6pm and you get voicemail. Call at 11am and you get a receptionist who's already on another line. Small businesses lose real revenue to unanswered phones, and the existing options are a phone tree nobody finishes or a 50$/hour answering service that can't actually book anything.

We wanted a receptionist that does the work — not one that takes a message.

What it does

Steve Assist answers your business phone. It knows your hours, your services, your pricing, and your staff. It answers questions the way a trained front-desk person would, and when a caller wants an appointment, it books one — the right provider, the right slot, written into the real calendar.

It's running today for marketing studio use cases. You can call it: +1-855-501-0082.

How we built it

A Rust/Axum backend bridges Twilio's media stream to a full-duplex pipeline:

Deepgram — speech-to-text
NVIDIA Nemotron 3.5 Lightning on Nebius Token Factory — reasoning and tool calls
ElevenLabs — speech synthesis
Firestore — storage for both configuration and conversation summaries

Scheduling happens through tool calls into the business's actual calendar system.

We chose Lightning specifically for its 30B MoE / 3B active-parameter profile. On a phone call the metric that matters is time-to-first-token, not benchmark depth — a caller forgives a simple answer but not a two-second silence.

The reasoning layer sits behind a provider-agnostic trait, so the model is swappable without touching the audio path or the scheduling logic. Nemotron is the default.

Challenges we ran into

Latency is the whole product. Parallelising the pre-greeting Firestore lookups cut the server-side setup from ~1–1.5 s to ~0.1 s, bringing the greeting pipeline to ~0.75 s (lookups 0.1 s, Nemotron 0.26 s, ElevenLabs 0.4 s); the caller hears the first word about 1.8 s after pickup, with the balance being network transit.

Tool-calling accuracy is not optional. A chat agent that hallucinates a detail is annoying. A receptionist that books the wrong provider costs the business a patient. We built an eval set of 50 real scheduling transcripts before trusting the model on live calls.

Prompts don't port. A system prompt tuned against one model underperforms on another in ways that look like model quality but are actually prompting. We ended up versioning prompts per provider.

Accomplishments that we're proud of

A real caller can't tell. That was the bar, and we're close. Two major factors for success in this direction are: 1. A continuous improvement on the service latency. 2. Using ElevenLabs for speech synthesis provides the cleanest and most human like voice over capabilities.

Beyond that: the model layer is genuinely pluggable, not pluggable-in-principle. Swapping providers is a config change, and both paths are tested. That turned out to matter more than we expected — data residency is a live objection in healthcare, and being able to say "your calls run on a dedicated endpoint in your region" changes the conversation.

What we learned

Where it was great

Speed. ~350 ms for a short conversational turn, ~500 ms median including tool-calling turns, ~260 ms for the greeting — from a 120B model, in the same region. That's the single biggest thing it brought: the call feels responsive, and it's what makes the 1.2 s caller-perceived turn latency possible.
Tool-calling mechanics are solid. Across every run it produced well-formed calls to check_availability, suggest_meeting_slots, book_meeting, end_call, with correct ISO timestamps and offsets, passed the new start_weekday and caller_confirmed arguments without being prompted twice, and recovered from tool errors (re-issuing with the corrected date). No malformed JSON, no hallucinated tool names.
Conversational quality is good enough for a phone persona. Natural greetings, used the caller's name, picked up the agent name change immediately, concise replies (median 39 output tokens), clean summaries and memory extraction. Callers wouldn't notice it isn't a frontier model in ordinary exchanges.
Responds to structure. Every time we moved a rule from prose into the tool contract (weekday check, two-step confirmation, already_booked, spoken dates in tool results), it followed the new shape on the next run. It's steerable through the API surface.

Where it's not so great

Date arithmetic. Even with the explicit 14-day "weekday → date" list in the prompt it still asked for Saturday Oct 10 as "Friday" in two of the last three booking runs. We fixed this by refusing or snapping on the server, not by prompting harder.
Rules in prose are followed loosely. It booked without the mandated read-back, used markdown bullets despite "never use markdown", insisted on an email address that the rules said was optional, and narrated its plan aloud. None of this was fixable reliably with prompt wording; each needed a code guard or text post-processing.
Reasoning leaks. With thinking disabled it still emitted a stray with its plan in front of it, which went to TTS until we stripped it.
Misreads tool results. "No slots on Saturday" became "Jeremy has nothing free Friday"; a booked result became "just to confirm…"; it probed slots one by one until it ran out of tool rounds instead of using the slot list it already had.
Variance. Four clean booking passes, then two different new failure modes with no relevant code change. Any single successful demo call proves less than it seems; the e2e harness run repeatedly is what tells you the truth.

What's next for Steve Assist

Deeper calendar integrations, so onboarding a business is a connection rather than a configuration. Multi-language, which the voice stack already supports and the market clearly wants. And per-tenant model selection as a product feature rather than a deployment detail — let each business choose where its calls are processed.