voxharness

Inspiration

Every voice-AI team I've watched tests their agent the same way: someone on the team calls their own number, listens, and decides "yeah, sounds fine." Every code change. By hand. In every language they support.

That doesn't scale, it isn't reproducible, and the number that actually makes real callers hang up — the dead air before the agent responds — never shows up in a unit test. Latency measured on a clean datacenter socket is not the number that matters. Latency measured over the phone, through jitter, codecs, and end-of-turn detection, is.

The 2026 voice-AI stack has good testing SaaS (Cekura, Hamming, Coval), but they're closed platforms you upload your agent to. There was no open, self-hostable harness that runs inside your own CI, over the real telephony path, with Indian languages as first-class citizens. So I built it.

What it does

voxharness turns the manual "call it and listen" test into one command:

voxharness dial --real --number +91XXXXXXXXXX --scenario examples/basic-hindi-kyc.yaml

voxharness is the caller. It plays each scripted customer line with Sarvam TTS, listens to your agent's reply with Sarvam STT, measures the latency between the two, checks for repetition, scores naturalness with Groq — then moves to the next turn. Your agent is the only thing on the other end. The only thing a human touches is the scorecard.

It scores three things that come straight from shipping multilingual voice in production:

1. Response latency — does the agent start answering within budget (target sub-1.4s)?
2. Repetition — does the agent ever ask the same question twice in one call?
3. Naturalness — how human vs. robotic does it sound (0–100, LLM-judged)?

Then it prints a scorecard and returns a non-zero exit code if a hard check regressed — so it fails your build the same way a broken test would. Add --json and you can post the scorecard on a PR.

Scenarios are small YAML files a non-programmer teammate can read and edit — with language: hindi | kannada | telugu | tamil | english as a first-class field, not an afterthought.

There's also a live web dashboard (voxharness-web.vercel.app) where you can run tests and watch them stream in real time: latency gauge, live transcript, animated scorecard.

How I built it

• Core harness (Python 3.10+): a turn loop that drives the conversation — TTS out, STT in, timers around the gap. Everything is driver-based, so the telephony layer, TTS, and STT are swappable.
• Mock drivers first. voxharness ships with mock drivers so you can run the entire pipeline end-to-end with zero credentials (--mock). That decision saved me: I could test the scorer, the YAML parser, and the CLI in CI without placing a single real call.
• Real mode adds actual telephony (a deployed media server + credentials) behind a --real flag and an optional install extra (pip install -e ".[real]"), so the core stays zero-setup.
• Scoring: latency is measured between end-of-user-audio and start-of-agent-audio; repetition uses a similarity threshold (configurable, default 0.85) against earlier agent turns; naturalness is scored by Groq against a minimum bar set per scenario.
• Web dashboard: Next.js 15 + Tailwind v4 + Framer Motion, streaming test progress in real time.
• CI: GitHub Actions runs the test suite on every push — the harness that tests agents is itself tested.

Challenges I ran into

• Measuring the right latency. "Time until the API responds" and "time until the caller hears a voice" are different numbers. Getting end-of-turn detection right over a phone codec — without cutting the agent off or padding the measurement — was the hardest part of the whole project.
• Repetition is fuzzy. Agents rarely repeat themselves verbatim; they rephrase. Exact-match detection catches nothing, so I use similarity scoring with a tunable threshold instead.
• Indic-language STT round-trips. Scoring a Hindi or Kannada conversation means the transcription itself has to be trustworthy first — a mis-transcribed reply would fail a healthy agent. Sarvam's Indic-first models were the unlock here.
• Designing for zero-credential first-run. Most infra tools fail at pip install. Making the mock path the default meant anyone can see a full scorecard in under a minute, which is also what makes it usable in CI without secrets.

Accomplishments that I'm proud of

• No human picks up a phone. The entire "call your own agent and listen" ritual is now one command with a pass/fail exit code.
• Indian languages are a first-class citizen — hindi, kannada, telugu, tamil are scenario-level fields, not a stretch goal.
• The mock-mode design: full end-to-end demo with zero API keys.
• It's genuinely open source (MIT) and self-hostable — the thing the closed SaaS platforms don't offer.
• A real-time web dashboard that makes the scorecard watchable, not just readable.

What I learned

• Test infrastructure has an adoption physics: the first run has to work with nothing configured. Mock-first isn't a demo trick, it's the product decision.
• Voice AI quality is a telephony problem as much as a model problem. The phone path (codecs, jitter, turn-taking) changes the numbers you get on a clean socket.
• LLM-as-judge works for naturalness, but only with a tight rubric and a per-scenario minimum bar — a raw "rate this 0–100" prompt drifts.

What's next for voxharness

• More telephony drivers (Twilio, Exotel, Plivo) so teams can use whatever carrier they already have.
• Barge-in and interruption testing — can the agent handle being talked over?
• A GitHub Action published to the Marketplace: uses: voxharness/dial@v1.
• Regression tracking across runs: latency trends per commit, not just per call.
• More Indic languages (Marathi, Bengali) as Sarvam's model coverage grows.

Built With

• framer-motion
• github
• groq
• next.js-15
• python
• sarvam-ai-(tts/stt)
• tailwind-v4
• twilio-style-telephony