StandBy

When your app and admin panel fail, Standby provides a recovery path maintained by GitLab Duo, tested in CI, and approved by you.

Comment

Inspiration

When an application breaks, its admin panel often breaks with it. The recovery tool you need may depend on the very system you’re trying to fix.

Even when an emergency script exists, there’s another question: does it still work with the latest release?

We built StandBy to make recovery part of the development workflow. Every release should have a recovery path that is maintained, reviewed, and tested before an incident happens.

What it does

StandBy is an independent recovery console that stays available during the configuration failure demonstrated in our app.

An operator can preview a repair, inspect its exact scope, and approve it once. StandBy checks the release and state revision, applies the bounded change, verifies the result, and records a recovery receipt.

Our demo demonstrates two actions:

  • Configuration recovery: restore valid configuration, then verify that both the application and its admin route respond successfully.
  • Queue recovery: release eligible expired leases while preserving active work and exhausted retries.

GitLab Duo helps maintain the recovery code, while GitLab CI checks that it works. The operator controls execution.

How we built it

We built two independent Node.js services on Google Cloud Run: the sample application and the StandBy recovery console. They share a dedicated Firestore database containing configuration, queue state, revisions, and recovery receipts.

The console has its own authentication and routes. Recovery runs directly against the shared state and does not require GitLab or Duo to be available during an incident.

GitLab provides the development and verification workflow:

  • A custom Duo Agent Platform recovery maintainer inspects release changes, updates repair code, and adds regression tests.
  • Merge requests and Duo Code Review make those changes reviewable before merging.
  • GitLab CI runs recovery drills and security checks before packaging.
  • CI Expert, Security Analyst, and Orbit contributed pipeline guidance, finding assessment, and investigation of repair evidence.

Approved source is built with Cloud Build, stored in Artifact Registry, and deployed to the separate Cloud Run services.

Challenges we ran into

The most important challenge was proving that recovery code could fall behind application code.

We introduced a version-two configuration format using a checksummed envelope. The existing repair could no longer read it, and the recovery drill failed. Duo updated the decoder and added regression tests while preserving version-one compatibility.

We also had to make approvals safe under changing state. A preview can become stale after another operator acts or the demo resets. We addressed this with expiring, session-bound, single-use approvals and transactional revision checks.

Testing the tests mattered too. Duo Code Review identified a stale-approval test that could pass for the wrong reason. We tightened the assertions and reran verification.

Accomplishments that we're proud of

We demonstrated a complete, traceable cycle: a release change broke recovery, CI caught it, Duo produced a repair, review improved it, and the deployed console successfully recovered the application.

The project includes 20 behavioral checks, plus live verification of recovery, persistence, stale approvals, replay refusal, and competing operators.

We’re especially proud that the demo shows actual agent activity, merge requests, review findings, and pipeline results. The evidence is available in the public repository.

What we learned

Recovery tools need the same engineering discipline as application code: versioned contracts, meaningful tests, review, and clear ownership.

We also learned that a successful database write is not proof of recovery. Independent verification—and an honest record when verification is incomplete—is essential.

AI assistance was most useful when given a scoped task with measurable acceptance criteria. The strongest result came from combining agent-generated changes, automated checks, and human review.

What's next for StandBy

Our next steps are to add isolated demo sessions, stronger operator identity and role-based access, immutable recovery snapshots, and support for more release contracts.

We also want to expand recovery drills and make it easier to package a verified recovery path alongside each release.

Today, StandBy demonstrates bounded recovery from configuration failures. Extending its resilience to infrastructure and database outages will require additional architecture and testing.

The goal is straightforward: when a release ships, its recovery path should ship with it.

Built With

Share this project:

Updates

Submission history