CallYourAgent

• 

• 

• 

• 

• 

Inspiration

AI agents are becoming capable of working autonomously for long periods of time, but there is still a fundamental gap between autonomous execution and human judgment.

When an agent reaches a decision that genuinely requires its owner, the usual experience is a notification, a dashboard, or a completely paused workflow. We wanted something much more natural: when your AI really needs you, it should be able to call you.

That became the idea behind CallYourAgent — turning the phone into a direct human escalation channel for autonomous agents.

What it does

CallYourAgent lets an AI agent escalate an important decision, clarification, or approval to its owner through CALL-E.

The key idea is that the entire agent does not have to stop. An escalation belongs to the affected task or branch, so unrelated work can continue while the human decision is pending.

The owner can also request a callback to an active agent to ask for progress, ask questions, or provide new instructions. During these interactions, the system uses the agent's current persisted context, stores the human response as structured state, and delivers instructions through a durable queue at a safe checkpoint.

This makes the phone call part of the agent's control loop rather than just another notification.

How we built it

We built CallYourAgent as a persistent agent control plane around the phone layer.

The backend manages agent registration, run and branch state, escalations, owner decisions, callback requests, instruction queues, checkpoints, audit history, call policies, idempotency, retries, timeouts, and recovery.

We expose the same core through HTTP APIs, MCP, and a TypeScript integration layer, making the system usable across different agent environments rather than tying it to a single AI provider.

CALL-E is isolated behind a provider abstraction. We also built a deterministic fake CALL-E provider so the complete system can be tested repeatedly without spending phone credits, while keeping a production provider path for real calls.

Challenges we ran into

The hardest part was not getting an AI to initiate a phone call. The difficult part was making the system reliable around a real-world side effect.

We had to handle situations such as a provider timing out after potentially accepting a call, preventing duplicate calls, recovering after restarts, preserving decisions, and ensuring that one blocked branch never accidentally freezes unrelated work.

We also had to be careful about what “interrupting an agent” actually means. Instead of pretending to inject instructions into an in-flight model generation, we designed explicit safe checkpoints where durable human input can be consumed reliably.

Accomplishments that we're proud of

We are proud that CallYourAgent is designed as a general integration layer, rather than a scripted demo built around one specific agent.

The core workflow is backed by persistent state and explicit state transitions, with branch-scoped blocking, durable human decisions, instruction queues, idempotent call orchestration, auditability, scoped credentials, and recovery semantics.

We are especially proud of the principle at the center of the project:

A human decision should block only the work that actually needs that decision.

That lets an agent remain meaningfully autonomous while still having a direct path to its owner when autonomy reaches its limits.

What we learned

We learned that human-in-the-loop AI is less about adding a “human approval button” and more about designing how human input becomes part of an autonomous system's state.

A phone interaction introduces uncertainty: calls can fail, providers can timeout, processes can restart, and the same request must never accidentally create multiple real-world side effects.

That forced us to think in terms of durable state, state machines, idempotency, scoped permissions, safe checkpoints, and recovery instead of relying only on conversational prompts.

We also learned that the most useful role for a human is not to constantly supervise an agent — it is to become available only when the agent genuinely needs human judgment.

What's next for CallYourAgent

The next step is to make CallYourAgent a common human-control layer for long-running agents across the ecosystem.

We want deeper integrations with Claude/Claude Code, Codex, ChatGPT-compatible workflows, and custom agents, along with richer agent context during calls and more powerful owner steering.

Longer term, we envision agents that can operate independently for hours or days while knowing that a real human is still reachable when something genuinely important falls outside their autonomy.

The goal is simple:

Don't make humans monitor autonomous agents. Give autonomous agents a reliable way to reach humans.

Built With

• agents
• ai
• api
• automation
• call-e
• hci
• llm
• mcp
• next.js
• node.js
• oauth
• sdk
• security
• sqlite
• supabase
• telephony
• typescript
• voice
• webhooks