Inspiration
Anyone who trains ML models knows the feeling: you start an 8-hour run, go to bed, and wake up to find it went NaN at minute 12. The GPU sat idle all night and you lost a day. Monitoring tools can show you a dashboard, but nobody watches dashboards at 3 a.m. We wanted something that acts more like a loyal teammate than a dashboard: it notices the problem, gets hold of you, explains it, and helps fix it.
We took the name from the Odyssey. Argos is Odysseus’s dog, who waits 20 years and is the first to recognize his master when he comes home. That’s the job we wanted our watcher to do: wait loyally, catch trouble first, and call you home.
What it does
Argos watches your ML training runs. When a run crashes, goes NaN, runs out of memory or stops improving, Argos:
- Detects it by tailing your training logs and metrics in real time.
- Diagnoses it with Grok, which finds the likely cause (for example, a learning rate that’s too high) and drafts a fix on a separate git branch.
- Calls you. Argos places a FaceTime call to your phone and shares your laptop screen live on the call: the code, the logs and the loss curve.
- Talks it through. A voice agent called Mentor explains what broke in plain words and suggests one fix. You can ask questions or push back, the way you would with a teammate.
- Fixes it once you say yes. When you approve out loud, Argos applies the fix, resumes from the last checkpoint, and confirms the loss is healthy again.
How we built it
- Watcher (Python): runs the training script, tails metrics.jsonl and the logs, and uses detectors to catch crashes, NaN or exploding loss, out-of-memory errors and plateaus. Each failure becomes a structured incident.
- Diagnosis (Grok): a Grok coding agent, guided by a custom skill file, reads the incident and the repo, explains the cause, judges how serious it is, and drafts a fix on an agent/fix- branch.
- Escalation MCP server: an MCP server lets the agent escalate to a human. It starts the call, publishes the screen share, and sends the human’s decision back to the agent.
- Phone bridge (Chert): connects iMessage and FaceTime to our LiveKit room, so a regular phone call reaches our agent.
- Live call (LiveKit + Gemini + ElevenLabs): our agent joins a LiveKit room and publishes a live screen-share track. On the call, speech-to-text feeds Gemini, which runs the conversation and calls tools (switch screens, apply the fix). ElevenLabs provides the voice.
- Fixer: applies the approved fix, resumes training from the checkpoint, and checks that the loss recovers.
- Mentor persona: the call agent speaks as a calm guide from the Odyssey, with a little epic flavor but never at the expense of accuracy.
In short: Grok writes the code, Gemini sees and talks.
Challenges we ran into
- Getting a real phone call into our system. Bridging FaceTime into a LiveKit room and publishing a live screen track that shows up on the phone took a lot of debugging. The LiveKit Python SDK also changes between versions.
- Making the call feel natural. Early versions went silent while applying the fix, then asked “are you still there?” and said goodbye twice. We redesigned the flow: a quick acknowledgment, a quiet fix, a clear “it’s fixed, anything else?”, and a single goodbye.
- Mac and Windows support. Our team uses both, and the demo first depended on Mac-only paths and bash scripts. We made it portable with a single Python launcher.
- Keeping the AI safe. An agent that edits your training code is risky. We added approval gates, separate fix branches, dry runs and full action logs so nothing destructive happens without a clear yes.
- Coordinating several AI services in real time. Grok, Gemini, ElevenLabs, LiveKit and Chert each have their own latency and quirks. Getting them to work as one conversation was hard.
Accomplishments that we're proud of
- A hands-off end-to-end demo: a training run fails, your phone rings, you see your own screen on the call, say “fix it”, and the run recovers.
- A voice agent that feels like a teammate on a call rather than a notification.
- The human stays in control. The AI only acts with explicit approval. It runs on both macOS and Windows.
What we learned
- How to build real-time voice agents with LiveKit, and how much latency and turn-taking shape the experience.
- How to use MCP to give an AI agent a clear, safe way to escalate to a human.
- How to split one job between models: Grok is strong at reading and fixing code, and Gemini is strong at live conversation and understanding what’s on screen.
- Small conversational details, like an acknowledgment or a single clean goodbye, change how trustworthy an agent feels.
What's next for Proactive Training Agent
More failure types: data-loader stalls, gradient explosions, divergence between training and validation loss, and stuck distributed jobs. Cloud and cluster support: watching runs on remote GPUs, Slurm and Kubernetes, not just a local laptop. Team escalation: call whoever is on call, and pass the issue to a teammate if you don’t pick up. Learning from past fixes: remembering what worked on earlier incidents to diagnose faster. Beyond ML: the same watch, call and fix loop could work for CI pipelines, data jobs and deployments.
