Runbook Drift

Old tickets remember how systems used to work. Runbook Drift keeps agents aligned with how they work now.

Comment

Inspiration

Enterprise agents learn from historical support tickets, but infrastructure changes faster than ticket history. We wanted to prevent agents from confidently recommending procedures that were correct years ago but dangerous today.

What it does

Runbook Drift analyzes 200 synthetic Jenkins tickets and migration events to identify four architecture eras. It generates a current recovery skill, preserves three exact legacy exceptions, validates the skill with deterministic replay, and publishes it to Firestore. When a Pub/Sub architecture event arrives, v4 is marked stale and a separately validated v5 becomes current.

How we built it

We built the backend with Python and FastAPI on Cloud Run. Gemini 3.5 Flash runs through Vertex AI and the Google GenAI SDK, with Pydantic enforcing structured outputs. Pub/Sub delivers architecture events, Firestore stores immutable skill versions, and Cloud Build with Artifact Registry handles deployment.

Challenges we ran into

The hardest problem was separating popular historical advice from currently valid advice. We also needed to support limited legacy exceptions without making old procedures valid everywhere. Finally, we had to make Gemini-generated skills deterministic enough to pass strict replay gates.

Accomplishments that we're proud of

Runbook Drift produces a portable, self-contained Markdown skill rather than only displaying an AI answer. Both the initial skill and its updated version reach 100% replay scores. The system preserves v4 as historical evidence while publishing v5 under a new immutable identity.

What we learned

Historical data needs temporal context. Migration events and retired actions are as important as successful past resolutions. We also learned that generated operational knowledge needs deterministic validation before it can be trusted.

What's next for Runbook Drift

We plan to connect real Jira, Git, Terraform, Cloud Asset Inventory, and deployment events. We also want to support more platforms, persistent user sessions, approval workflows, and automatic drift monitoring across complete enterprise runbook libraries.

Built With

Share this project:

Updates

Submission history