AgentTT

Inspiration

We were inspired by a simple question:

What if calling an AI felt exactly like calling a real person?

Although AI assistants are becoming increasingly capable, most interactions still happen through chat windows or voice apps. Traditional phone calls remain the most natural communication method for millions of people, yet they have barely changed in decades.

We wanted to bridge the gap between modern AI agents and the familiar experience of a phone call, allowing anyone to simply pick up a telephone, dial a number, and have a natural conversation with an intelligent AI assistant.

What it does

AgentTT transforms a traditional telephone into an AI-powered communication device.

Users can call the AI through a SIP phone or a regular telephone gateway and interact using natural speech. The system supports real-time speech recognition, streaming LLM responses, text-to-speech synthesis, and intelligent tool calling.

Beyond simple conversations, AgentTT can:

Answer questions naturally in real time.
Execute tasks through AI agents.
Connect with external tools and APIs.
Maintain conversational context across turns.
Deliver low-latency, human-like voice interactions.

The goal is to make AI accessible through the most universal interface ever invented—the telephone.

How we built it

We built AgentTT around a real-time voice agent architecture.

The system uses SIP telephony to receive phone calls, streams audio through automatic speech recognition (ASR), sends transcripts to a large language model for reasoning and tool execution, and converts responses back into natural speech using text-to-speech (TTS).

Our architecture includes:

SIP-based telephony integration
Streaming ASR and TTS pipeline
Large Language Models for reasoning
Agent framework for planning and tool execution
Real-time audio streaming with low latency
Backend services for session management and conversation memory

The entire pipeline is optimized to minimize latency so conversations feel fluid and natural.

Challenges we ran into

One of the biggest challenges was reducing end-to-end latency.

Unlike text chat, phone conversations require responses within a few hundred milliseconds to feel natural. We optimized streaming audio processing, incremental speech recognition, LLM response generation, and speech synthesis to reduce waiting time.

Another challenge was handling interruptions and turn-taking. People naturally interrupt during phone calls, so the system needed to detect barge-ins, stop speech playback gracefully, and continue the conversation without losing context.

Integrating traditional telephony with modern AI agent frameworks also required careful engineering across networking, audio streaming, and state management.

Accomplishments that we're proud of

We are proud that AgentTT makes AI accessible through hardware that has existed for decades.

Instead of requiring users to install an app or learn a new interface, anyone can simply make a phone call and interact with an intelligent AI agent.

We're also proud of achieving smooth real-time conversations with low latency while integrating multiple AI components—including speech recognition, language models, speech synthesis, and agent workflows—into one seamless experience.

What we learned

Building AgentTT taught us that creating a great voice experience is about much more than connecting an LLM to speech.

Real-world voice agents require careful optimization of latency, interruption handling, conversation memory, telephony protocols, and user experience. We also learned how important streaming architectures are for making AI feel responsive and natural.

Most importantly, we realized that the best AI interfaces often disappear into familiar human interactions instead of introducing new ones.