PM Autopilot

Built on Novus.ai — PM Autopilot turns a metric drop into a merged PR, autonomously.

Comment

Inspiration

Every PM has lived this story. A metric drops at 2am. You get paged. You open four dashboards, scrub through session recordings, file a Jira ticket, wait for sprint planning, and two weeks later the bug is fixed — if it doesn't get deprioritized first.

The real problem isn't that PMs lack tools. It's that the tools don't talk to each other. Novus.ai knows what broke and which users it broke for. GitHub knows where the code is. Gemini knows how to fix it. But connecting those three things — in the right order, with the right context — takes a human 2–3 hours every single time.

We asked: what if that entire chain ran autonomously, powered by Novus.ai's product intelligence data, so the PM only had to make one decision?

What it does

Built on Novus.ai — PM Autopilot watches your analytics, reads your session replays, finds the bug in your code, and opens the fix. Your only job is the approve button.

PM Autopilot runs an 8-step autonomous pipeline the moment a metric anomaly is detected:

  1. Detect — Calls Novus.ai's getArtifactMetrics to confirm the drop is real (not noise), validating against a configurable threshold and minimum visitor count
  2. Diagnose — Uses getRelatedArtifacts to find which child feature of the affected page has the steepest drop, narrowing "Cart Page broke" to "CouponInput broke"
  3. Scan session replays — Calls listReplays filtered to rage-click frustration signals, fetches event sequences via getSessionReplayEvents, and sends them to Gemini 2.0 Flash: "What's the common failure pattern across these 5 sessions?"
  4. Locate code — Uses the Novus.ai track event's currentCode.filePath — a direct pointer to the source file — so no GitHub search is needed. Falls back to CSS selector search if it's a feature artifact.
  5. Generate fix — Sends the file content + failure pattern to Gemini 2.0 Flash, which returns a minimal unified diff targeting exactly the broken lines
  6. Open PR — Creates a branch, commits the patched file, and opens a pull request with the root cause, session count, and run ID in the PR body
  7. Deploy in-app guide — Calls Novus.ai's createArtifact to place a temporary banner on the affected page: "We noticed an issue and a fix is in review" — so live users aren't left confused while the PR is open
  8. Notify Slack — Posts a structured message with the metric, root cause, PR link, and sessions analyzed

The PM opens their dashboard, sees a run in "Awaiting approval", reads the diff, and clicks Merge or Reject. That's it.

How we built it

The core insight was that Novus.ai's MCP server is the connective tissue between all three systems — not just a passive analytics viewer.

The Novus.ai integration goes deep:

  • Authentication uses OAuth 2.1 client_credentials against novus-api.pendo.io/mcp-auth/token, with a 24-hour token cache and automatic 401 retry
  • All MCP calls use JSON-RPC 2.0 over POST /mcp with Accept: application/json, text/event-stream — the SSE transport that Novus uses
  • We bypass Next.js 14's patched fetch() entirely (it breaks SSE streaming) and use raw node:https for all Novus calls — a non-obvious fix that cost hours to diagnose
  • getArtifact on a track event returns data.currentCode.filePath — a direct source file reference. This was the "unlock" moment: instead of guessing which file to look at, Novus tells us exactly where the code is

The stack:

  • Next.js 14 App Router (TypeScript, Tailwind CSS)
  • SQLite with WAL mode for concurrent-safe run persistence
  • Gemini 2.0 Flash via OpenAI-compatible endpoint (generativelanguage.googleapis.com/v1beta/openai/)
  • GitHub REST API via Octokit for branch creation, file commits, and PR management
  • Server-Sent Events for real-time step streaming to the dashboard

The locate step uses a three-tier precision hierarchy: track event file path (zero search) → feature CSS selector (targeted GitHub search) → replay element text (LLM-ranked search). Each tier degrades gracefully — a failed locate skips to the next step rather than failing the whole run.

Challenges we ran into

Next.js 14 breaks SSE responses. Next.js patches the global fetch() at startup to add caching semantics. When that wrapper receives an SSE response, it tries to buffer and cache the stream — which silently fails with fetch failed on Node 22. The fix was writing a raw node:https client that bypasses the wrapper entirely. This took the better part of a day to diagnose because the error message gave no hint about the cause.

SQLite SQLITE_BUSY under concurrent writes. When a background agent run was writing to the database while a "Clear all runs" request came in, SQLite threw SQLITE_BUSY immediately. The fix was enabling WAL mode (PRAGMA journal_mode = WAL) and setting PRAGMA busy_timeout = 5000 — now writers queue for 5 seconds instead of failing.

The Novus artifacts endpoint doesn't exist. We initially assumed there was a REST endpoint like GET /mcp/artifacts. There isn't. Novus uses JSON-RPC 2.0 over a single POST /mcp endpoint, and even that requires Accept: application/json, text/event-stream or it returns 406 Not Acceptable. Learning the exact transport protocol required live-testing against the API with PowerShell.

React StrictMode double-mount causing double Novus API calls. The modal's artifact fetch was firing twice in development, triggering two parallel MCP requests. The fix was an AbortController with a module-level promise cache that coalesces concurrent requests onto a single in-flight fetch — a pattern borrowed from SWR/React Query's deduplication logic.

Metric tab state management. The six-metric tab bar had a subtle bug where clicking a tab would highlight it but show the previous metric's data. The cause: the server's metricKey response (which might be a fallback key) was being synced back into selectedMetricKey state via setSelectedMetricKey() inside the .then() handler — which re-triggered the useEffect (since selectedMetricKey was in its dep array), which fired another fetch, which set metricState({ phase: 'loading' }), wiping the just-loaded data. The fix: all display values read from allMetrics[] keyed by selectedMetricKey directly — the server's 6-metric summary is used for instant tab switching, no re-fetch needed.

Accomplishments that we're proud of

Novus.ai is the brain, not just the data source. .We used it as an active participant — it tells the agent what's broken (getArtifactMetrics), which specific element caused it (getRelatedArtifacts with CSS selectors), how users behave around it (listReplays, getSessionReplayEvents), exactly where the code is (getArtifact → currentCode.filePath), and how to talk to users while the fix is in review (createArtifact type=guide). That's five distinct MCP tools working together as an autonomous decision chain.

Zero-search file location. The getArtifact tool returns data.currentCode.filePath for track events — a direct pointer to the source file. When this works, the locate step is deterministic: no GitHub search, no LLM ranking, no guesswork. Novus already knows where the code is because it was instrumented there.

The UI degrades gracefully under real conditions. Every external call has explicit error states, not silent fallbacks. When Novus returns no metric data for a feature (because avgTime is page-only), the UI falls back to the first available metric and shows a note explaining why. When a CSS selector search returns zero results, the run continues and the fix step generates a patch from the pattern description alone. Nothing silently swallows errors.

The full pipeline actually runs end-to-end. the real orchestrator runs, real Novus MCP calls fire, a real GitHub PR gets opened.

What we learned

Product intelligence data is deeply underutilized. Novus.ai already knows the CSS selector of every instrumented element, the source file of every track event, the user behavior sequence leading to every abandonment. This data is rich enough to drive autonomous engineering decisions — not just dashboards.

Autonomous agents need graceful degradation more than perfect accuracy. An agent that fails completely when one step doesn't work is brittle. PM Autopilot's locate step has three fallback tiers; the fix step works without a file reference; the guide and notify steps run even if the PR step skips. The goal is always to leave the situation better than it started, even partially.

The PM's time is the bottleneck, not the developer's. The 2–3 hours of triage before a bug gets to a developer is where the real cost is. Cutting that to zero doesn't require perfect automated fixes — it requires getting a correctly-scoped, well-contextualized pull request in front of a human as fast as possible.

Token minting is not trivial at scale. The Novus OAuth 2.1 flow mints 24-hour tokens that need to be cached and refreshed. Getting that right — with a module-level cache, 5-minute early-expiry buffer, and automatic 401 retry — is infrastructure work that has to be done correctly before anything else works.

What's next for PM Autopilot

Webhook-triggered runs at scale. The /api/webhook endpoint already exists with HMAC-SHA256 signature verification. The next step is connecting it directly to Novus.ai's signal feed so runs trigger automatically when anomalies are detected — no human needed to start the cycle.

Confidence scoring before PR creation. Before opening a PR, the agent should assess how confident it is in the fix — based on replay consistency, file match quality, and diff size. Low-confidence runs would flag for human review before creating the branch.

Multi-file fixes. The current fix step patches a single file. Many bugs span two or three files. The next version would use the locate step's candidate list to generate coordinated patches across related files.

Novus Signals integration. Once the Novus.ai signal feed matures, PM Autopilot would consume it directly — treating each signal as a trigger event, running the full pipeline, and surfacing only the runs that reach awaiting_approval to the PM.

Learning from approvals and rejections. Every time a PM rejects a fix, that's a training signal. Over time, the agent should get better at knowing when it has enough confidence to open a PR versus when to surface the diagnosis and wait for guidance.

Built With

  • nextjs
  • novus.ai
Share this project:

Updates