Mercury

Fire TV incident triage: one temporal candidate, or why not.

Comment

What it does

On Fire TV, Mercury names a deploy to the on-call engineer only when exactly one runtime change landed in the window, and otherwise says why.

The remote drives the journey: ask what changed, inspect evidence, stage the recorded demo action, confirm once and publish a responder handoff. A named commit is a temporal candidate, not proof of cause. No candidates, several candidates or insufficient data produce an explicit refusal.

The hosted board has eight labelled rows. Five demo services use offline fixtures to demonstrate the possible answers. Three LIVE rows read CloudWatch metrics for our own Lambda functions. Those functions have no connected deployment feed, so their errors are not attributed to commits and cannot trigger a rollback.

Inspiration

During an incident, the first question is often "what changed?" A shared board can help a room follow the same evidence, but a confident guess can send responders toward the wrong rollback. Mercury explores a narrower answer: one temporal candidate, or a clear reason it cannot name one. We have not yet tested the shared-screen premise with responders.

How we built it

The correlator compares errors with a pre-spike baseline, then counts runtime changes in the 75 minutes before the onset. On our synthetic held-out corpus it made 58 namings at 50.0% precision and found 4.8% of regressions. It made 29 wrong namings, against 482 for the latest-deploy baseline, which found more causes. These are synthetic results, not production accuracy or reduced incident time. With a 180-minute delay assumption, precision falls to 27.3%.

The Fire OS app is a Kotlin Leanback WebView host with a bounded D-pad bridge. BACK asks the page first, then history, then exits. Version 1.0.1 adds native, focused Try again recovery for failed main-document loads. Agent results and errors use an initially quiet, polite status region.

Python and FastAPI run on AWS Lambda. DynamoDB stores handoffs and single-use governance state. CloudWatch supplies the live rows. The spend panel is a labelled fixture. Optional Ask the agent requests an advisory Claude Sonnet 5 brief on Amazon Bedrock; the card identifies model-written fields or explains a deterministic fallback. Nothing requests a model on load or row selection. The cap is 40 attempts per warm process, not a global budget. The successful model-answer receipt is historical; this release's verification made no model call.

The 10-tool MCP 2025-11-25 endpoint supports a handshake and agent_status check without a model request. Stage binds an HMAC token to the exact action; Confirm checks its binding, lease, current recommendation and single use. The public cockpit records the demo decision only. No deployment pipeline is contacted and no real GitHub rollback dispatcher has been fired.

Challenges and what we learned

TV input and web callbacks have different timing. We had to handle BACK asynchronously, preserve remote focus, distinguish main-document failures from failed subresources, and provide recovery even when the page never loaded. Our first result cards also needed explicit screen-reader status announcements.

Synthetic precision is conditional on the corpus and delay assumptions. A refusal is not evidence that no deploy caused an incident. The next useful test is replaying real, already-investigated incidents with responders and measuring whether a candidate or refusal helps their first ten minutes.

Demonstration and limits

Android TV run 39961752709 independently installed the exact Fire OS APK 1.0.1, launched it normally and completed the online D-pad journey against hosted build a438bf1. The public APK and checksum are at: https://github.com/upgradedev/mercury_amazon/releases/tag/firetv-v1.0.1

The 163.09-second demo video is historical: its first 23.32 seconds show an earlier Android TV emulator build; the remainder is desktop-browser footage at f7512bf. It does not show a438bf1, the new recovery/status fixes, a Mac/Vega session or a physical Fire TV.

Native offline recovery has JVM lifecycle and Activity wiring checks. Physical-remote offline behavior and real screen-reader users remain untested. The optional React Native for Vega shell builds in CI, but we have no successful Mac/Vega runtime evidence.

Open source

Mercury and the extracted navigator are public and MIT-licensed: https://github.com/upgradedev/mercury_amazon https://github.com/upgradedev/firetv-spatial-navigation

Merged contribution https://github.com/upgradedev/firetv-spatial-navigation/pull/1 adds legacy key handling, collision-safe/restorable bridge cleanup, stricter navigability checks, tests and TypeScript declarations. Mercury currently embeds its own navigator; we do not claim an npm-published dependency or Vega validation of the extracted package.

Built With

Share this project:

Updates

Submission history