Inspiration
Agents are blind to time. An LLM has no clock — it can't feel 20 minutes pass, can't wake itself up, and dies with its session. So the moment a task involves time — "do X, then wait, then do Y before the window closes" — agents either block uselessly or bluff. We asked: what would it take to give an agent real time context — the ability to schedule its own future, keep multiple clocks running in parallel, and get woken up at the right moment even if the user's connection died? Timeline is that missing layer, demoed on the most time-dense tasks people actually do hands-free: recipes, workouts, lab protocols.
What it does
Timeline gives a voice agent a durable sense of time.
When the agent faces a multi-step task, it classifies each step: an instruction (say it, advance), a blocking step (wait for the human), or a timed step — where it arms a timer and keeps going, opening parallel tracks instead of standing still. A timer isn't a countdown: it's a structured spec of pings at absolute timestamps ("flip it" at T+10, "2 min left" at T+18, "take it out" at T+20), each tagged with a priority the agent assigned at creation. Critical pings barge in and interrupt the agent mid-sentence; normal pings queue for a natural pause, with a defer_max so queued never silently means dropped.
The clocks don't live in the conversation — they live in a sandbox that keeps ticking when the voice connection drops. The agent can reconnect and ask its own time layer "what's still running?" and get the truth instead of hallucinating it.
How we built it
Three services, split along one principle — the sandbox rings the bell; the agent decides what it means:
- ElevenLabs Agents runs the whole voice loop — Scribe v2 Realtime STT, the agent LLM (claude-sonnet-4-5), streaming TTS, barge-in — over one WebSocket. The agent gets time as tools: create_timer to schedule its future, check_timers to query it; fired pings flow back as contextual updates it must react to.
- Daytona is the durable time layer: a sandbox (--auto-stop 0) running a tiny wake loop — sleep to the nearest ping, emit, repeat — over one merged list of pings sorted by absolute fire_at. Never countdowns: absolute time is what makes state re-derivable from the wall clock and makes concurrent timers trivial.
- Braintrust traces every step decision, tool call, and ping (including measured ping lag), and powers a deterministic eval harness that replays agent turns against the live prompt and live tool schemas with a pinned clock.
Built strictly test-driven in seven slices — timer store → wake loop with an injected clock → tool surface → Daytona wiring → voice wiring — red test, minimum code, green. The core is ~800 lines of Python. CodeRabbit reviewed the PRs.
Challenges we ran into
- LLMs cannot do time arithmetic. Our first eval scored 0% on absolute timestamps — the model can't convert "20 minutes from now" into an epoch in its head. Fix: inject {{start_epoch}} at session start and make the prompt compute every ping from it. 0% → 100%.
- The agent's mental model of time kept being wrong in ways only live sessions exposed. It classified user-active durations (planks) as "wait for the human" instead of "I own this clock," armed zero timers, then hallucinated that a rest had finished. Each failure became a prompt rule ("any step with a stated duration is TIMED — you do the timing") plus a regression eval case.
- daytona exec doesn't forward stdin and buffers stdout until exit — which silently killed our ping stream: timers fired in the sandbox but nothing ever reached the agent, so it bluffed timer status. Rebuilt the transport on JSON-lines files plus a polling pump.
- The usual tooling parade: agents CLI crashing without a TTY, camelCase schema mangling causing 422s, a SIGPIPE mid-push creating duplicate remote tools.
Accomplishments that we're proud of
- 100% across all eval experiments — timer-spec quality (including concurrent 3-timer scenarios), time-query answering, and ping-reaction — up from a 0% baseline.
- The durability invariant holds, verified end-to-end: pings fire in the sandbox on schedule with no client connected, and the runner stays alive for timers that arrive after the store drains.
- Time as data, not vibes: priority and deadlines are attached at spec time by the LLM, so fire-time is a dumb deterministic branch — no hot-path model calls deciding whether to interrupt you.
- Every live failure became a deterministic eval case — the suite is a regression net, not a demo prop.
What we learned
- Absolute timestamps beat countdowns everywhere. One rule made concurrency trivial (one sorted list), durability real (re-derive everything from the wall clock), and gave us our most important eval scorer.
- Agents don't fail at time loudly — they bluff. A missing clock doesn't throw an error; the model confidently invents "your rest just ended." Giving agents time context means giving them a queryable source of truth and the discipline to use it.
- Evals turn prompting into engineering. "The model won't emit epochs" went from a vibe to a measured 0%, to a hypothesis, to a fix, to 100%, in four runs.
- Offline evals can't find everything: all three live sessions surfaced failure modes (misclassified steps, silent ping loss) no replay predicted.
What's next for Timeline
- Time context that outlives the sandbox: state on a mounted volume, so a stopped sandbox restarts and re-derives every timer from absolute time — overdue pings fire immediately, or drop deterministically if their window closed.
- Longer horizons: tasks whose clocks span hours or days (a deploy bake time, a sourdough, a follow-up window), where the agent that armed the timer isn't the session that hears it fire.
- Generalizing the layer: create_timer / check_timers as a drop-in time-context toolkit for any agent — voice or not — plus richer interrupt policy learned from the eval traces.
Log in or sign up for Devpost to join the conversation.