RunPulse

Inspiration AI researchers often launch training jobs on remote GPU servers and monitor them through a mix of terminal logs, nvidia-smi, and occasional manual checks. When a run diverges, stalls, or silently dies, the problem may remain unnoticed for hours—wasting compute, money, and research time. Existing experiment trackers are useful for visualization, but they still require researchers to notice the problem and interpret scattered evidence themselves. We wanted something closer to an on-call engineer for model training: a lightweight tool that watches an existing SSH-based workflow, detects trouble early, and explains what happened with evidence the researcher can verify. That idea became RunPulse. What it does RunPulse is an evidence-first incident assistant for remote AI training runs. It connects to a researcher’s server through their existing SSH configuration and monitors: Training logs and metrics Process health GPU utilization and memory The experiment configuration RunPulse uses deterministic detectors to identify three costly failure modes: Numerical failures such as divergence and NaN losses Training stagnation while the process and GPU remain active System failures such as a missing process, stale logs, or unexpectedly idle GPUs When an anomaly is detected, RunPulse freezes the relevant metrics, logs, configuration, process state, and GPU observations into an immutable evidence bundle. GPT‑5.6 then produces a structured diagnosis containing likely causes, confidence levels, recommended actions, safe verification commands, and explicit citations such as [E1] and [E2]. A local web interface presents the current run, incident timeline, evidence, diagnosis, and follow-up questions. RunPulse remains read-only: it never stops a job or executes a suggested command without the researcher. How we built it We built RunPulse as a local Python application with clearly separated components: A replay or read-only SSH source collects observations A normalizer converts logs and system snapshots into typed events Independent state-machine detectors identify numerical, stagnation, and system anomalies An incident builder creates a bounded, redacted evidence snapshot SQLite stores incidents, diagnoses, source offsets, and follow-up exchanges GPT‑5.6 generates schema-validated diagnoses through the OpenAI Responses API FastAPI, server-rendered templates, SSE, and lightweight JavaScript power the local dashboard A key architectural decision was to separate detection from explanation. Deterministic logic decides whether an incident exists; GPT‑5.6 explains the evidence but cannot create, dismiss, or change the severity of an alert. If the model is unavailable, the underlying alert and evidence remain usable. We used Codex throughout the project to refine the product scope, design component boundaries, implement detectors test-first, create replay fixtures, review failure handling, and prepare the submission workflow. Challenges we ran into The first challenge was avoiding false confidence. Sending an entire training log to a model can produce a persuasive explanation, but not necessarily a trustworthy one. We addressed this by constructing small, immutable evidence bundles and requiring every diagnosis to cite evidence identifiers that RunPulse validates. Another challenge was distinguishing infrastructure failure from monitoring failure. An SSH disconnection must not be reported as a dead training process. RunPulse therefore tracks connection health separately and suppresses system conclusions when remote observations are unavailable. Training logs also vary widely between projects. Automatically understanding every format would have expanded the scope dramatically, so the MVP uses explicit metric patterns and reports parser degradation instead of silently guessing. Finally, a live training failure is difficult to reproduce during a three-minute demo. We created deterministic, accelerated replay sources that use the same observation interface as real SSH monitoring. Accomplishments that we're proud of We are especially proud that RunPulse is more than an LLM log summarizer. Its most important qualities are: Every important diagnosis is linked to inspectable evidence Detection continues when GPT‑5.6 is unavailable SSH monitoring is read-only and uses the researcher’s existing setup Connection failures are separated from training failures Replay mode makes incidents reproducible and testable The product focuses on one complete workflow instead of becoming another general-purpose monitoring platform We are also proud of the human-in-the-loop boundary. RunPulse can recommend what to inspect, but the researcher remains responsible for actions that could affect a running experiment. What we learned We learned that combining deterministic systems with language models is often more useful than asking either one to solve the entire problem. Rules and state machines are effective at recognizing known failure conditions consistently. GPT‑5.6 is more valuable after detection, when it can connect metrics, configuration, logs, and system state into a readable explanation. We also learned that evidence design matters as much as prompt design. A compact, well-structured snapshot produces more reliable diagnoses than a large amount of loosely related context. Finally, graceful degradation is a product feature—not just an engineering detail. An AI-assisted tool should remain understandable and useful when the model, network, parser, or remote connection fails. What's next for RunPulse Next, we want to expand RunPulse carefully without losing its evidence-first design: Monitor multiple experiments and servers Add adapters for TensorBoard and Weights & Biases Learn experiment-specific baselines to improve anomaly detection Compare related runs and identify configuration changes associated with regressions Deliver notifications through Slack, Telegram, or email Allow optional, explicitly approved actions such as preserving a checkpoint or stopping a failed run Support team-level incident history and reusable diagnostic playbooks Our longer-term goal is to make RunPulse a reliable research-operations agent: one that helps researchers spend less time babysitting training jobs and more time understanding their experiments.

Built With

• api
• codex
• fastapi
• gpt-5.6
• openai
• openssh
• python
• responses
• sqlite