SignalPM

An AI Copilot For Product Managers

Comment

Inspiration

SignalPM came from one core PM problem: the most important product decisions are usually made with fragmented evidence. Support tickets say one thing, sales calls say another, NPS comments add nuance, and analytics signals lag behind. Teams still have to manually stitch all of this together while ARR risk is moving in real time.

The product idea was to make evidence-to-decision flow operational, not just analytical. We wanted a system that can:

  1. Ingest and normalize messy multi-source customer truth.
  2. Detect and rank pain clusters by business impact, not just mention count.
  3. Generate execution artifacts such as proposal, PRD, design handoff, and engineering handoff.
  4. Keep answers citation-grounded and auditable.
  5. Move from insight to action with Jira, Slack, and Email dispatch plus decision approvals.

The second inspiration was technical: simple one-shot RAG is usually not enough for PM-grade answers. So we built around iterative retrieval, sufficiency checks, and fallback modes that keep the system useful even when live agent runtime is slow or partially unavailable.

What it does

SignalPM is a full-stack PM intelligence system with four connected loops.

1) Evidence loop

  1. Pulls synthetic or uploaded multi-source data into signalpm_signals.
  2. Normalizes each record with customer metadata, extracted signal fields, provenance, and dedupe keys.
  3. Supports filtering and search across signals, clusters, proposals, PRDs, and memory.

2) Agent loop

  1. Routes user intent into workflows: synthesize, propose, spec, ask, design_handoff, engineering_handoff.
  2. Runs iterative evidence gathering with stop conditions: sufficiency met, diminishing returns, or duration budget reached.
  3. Validates grounding and optional workspace guard constraints.
  4. Persists structured outputs into clusters, proposals, PRDs, handoffs, and decision docs.

3) Decision loop

  1. Scores proposals and PRDs with explicit weighted factors.
  2. Stores decision ledger entries such as pending_review, approved, rejected with scoring breakdown and evidence lineage.
  3. Writes decision memory for future runs so later agents can leverage prior governance context.

4) Execution loop

  1. Runs Kibana-native multi-step workflows such as synthesis to proposal to spec, proposal to design, proposal to engineering, and war-room brief.
  2. Syncs workflow runs into app history.
  3. Provides step-level output inspection and dispatch actions for Jira, Slack, and Email.

How we built it

Architecture and runtime

  1. Backend is FastAPI with 64 route handlers across agents, ingestion, data views, analytics, decisions, actions, workflows, jobs, sources, uploads, system, and chat threads.
  2. Runtime policy enforces connector consistency at startup through KIBANA_CONNECTOR_ID and CHAT_DEFAULT_CONNECTOR_ID alignment.
  3. Orchestration is centralized in Orchestrator, which maps workflow intent to specialized agents and controls retries, timeouts, and fallbacks.
  4. Agent execution is unified through AgentRunner, handling loop control, citation extraction, sufficiency checks, grounding checks, persistence, and memory append.
  5. Search and retrieval use a hybrid strategy in QueryService: retriever path when available, bool fallback, then legacy in-memory scoring fallback.

Reliability engineering choices

  1. Retry-aware Kibana API clients handle transient 408, 425, 429, and 5xx classes.
  2. Circuit breaker opens after repeated Kibana failures and forces Elasticsearch-grounded fallback answers for a cooling window.
  3. Workflow wall-clock is bounded to avoid hanging UX.
  4. Chat routes implement fast and deep mode, agent fallback, and MCP fallback with timeout budgets.
  5. War-room report generation has deterministic fallback if narrative agent generation fails.

Data model and persistence

  1. Signals go to signalpm_signals.
  2. Cluster outputs go to signalpm_clusters.
  3. Proposal outputs go to signalpm_proposals.
  4. PRD outputs go to signalpm_prds.
  5. Handoff outputs go to signalpm_handoffs.
  6. Decision docs go to signalpm_decisions.
  7. Memory docs go to signalpm_memory.
  8. Workflow run history goes to signalpm_workflow_runs.
  9. Action previews and events are persisted for traceability.

Synthetic data and quality gates

  1. Deterministic generation currently produces 10,080 signals across 8 sources.

  2. Source split:

  • intercom 2160
  • salesforce 1680
  • gong 1440
  • slack 1200
  • delighted 1200
  • amplitude 960
  • google_meet 720
  • microsoft_teams 720
  1. Quality report generated February 27, 2026 passes with:
  • duplicate_rate: 0.0
  • numeric_field_coverage: 1.0
  • invalid_numeric_rate: 0.0
  • backward_compatibility_rate: 1.0
  • enterprise_coverage: 0.398413
  • unique_text_rate: 0.52996

Frontend and operator experience

  1. Next.js App Router shell unifies evidence, signals, trends, proposals, PRDs, sources, war-room, and chat.
  2. War-room page adds scenario simulation and strategic brief generation.
  3. Workflow console supports run sync by execution ID, search, phase-by-phase inspection, and dispatch actions.
  4. Chat API supports SSE stream events such as assistant_delta, tool_progress, completed, and error with multi-mode orchestration.

Agents in depth

Two orchestration layers exist: backend orchestrator runtime and 8 provisioned Agent Builder agents.

Backend Orchestrator runtime

  1. Classifies intent using keyword mapping and explicit workflow field.
  2. Applies workflow-specific iteration caps.
  3. Enriches context with workflow-aware memory layers.
  4. Executes through Kibana converse with bounded retry policy.
  5. On timeout or circuit open, builds fallback answer from Elasticsearch search results.
  6. Supports chaining such as synthesis to proposal to spec and top-3 auto generation branch.
  7. Builds handoff prompts from upstream structured outputs and citations.

Agent 1: Synthesis Agent (signalpm_synthesis_agent)

Purpose: detect and rank pain clusters with ARR-aware impact framing.

Key tools:

  • pain_point_aggregation
  • sentiment_by_theme
  • segment_analysis
  • competitor_mentions
  • revenue_impact
  • semantic_signal_search

Output contract includes structured impact metrics, ARR totals, segments, trend direction, confidence, and top signal references.

Agent 2: Proposal Agent (signalpm_proposal_agent)

Purpose: convert cluster pain into prioritizable proposals.

Computes score, urgency, strategic fit, and decision lineage. Persists proposal and paired decision document.

Agent 3: Spec Agent (signalpm_spec_agent)

Purpose: generate structured PRD drafts from proposal context and memory.

Normalizes required sections, computes completeness meta, and persists PRD plus decision record.

Agent 4: Evidence Chat Agent (signalpm_evidence_chat_agent)

Purpose: high-depth evidence-grounded Q&A with broad tool access and enforced grounding checks.

Agent 5: Evidence Chat Fast Agent (signalpm_evidence_chat_fast_agent)

Purpose: low-latency demo and triage responses with constrained tool usage.

Agent 6: Design Handoff Agent (signalpm_design_handoff_agent)

Purpose: production-ready design handoffs including flows, IA, states, accessibility, telemetry, and acceptance criteria.

Agent 7: Engineering Handoff Agent (signalpm_engineering_handoff_agent)

Purpose: implementation-ready plans including backend changes, API contracts, data models, migration strategy, rollout plan, and QA matrix.

Agent 8: Workflow Orchestrator Agent (signalpm_workflow_orchestrator_agent)

Purpose: invoke Kibana workflows and manage async execution state, retries, and dispatch actions.

Tools in depth

Includes 15 tools covering ES|QL aggregations, semantic search, memory retrieval, multi-step workflows, and dispatch integrations for Jira, Slack, and Email.

Challenges

  1. Live runtime reliability pressure under some modes.

  2. Benchmark artifact dated February 26, 2026 shows overall gate failures:

  • weighted score 0.52 out of 5.0
  • hard fails 71
  • live quality report weighted score 0.23 out of 5.0
  • timeout rate 0.875

  1. Maintaining strict grounding while preserving low latency required layered fallbacks.

  2. Workflow and webhook branching increased complexity.

  3. Current local test status February 27, 2026: 101 passed, 4 failed.

  4. Three failures are async pytest plugin configuration related.

  5. One failure is slack workflow fallback behavior mismatch.

  6. Keeping prompt contracts and runtime behavior synchronized required careful version discipline.

Accomplishments

  1. Built a complete operational AI PM loop, not just a chatbot.
  2. Implemented 8 agents, 15 tools, and 7 workflows in one coherent stack.
  3. Added resilience patterns including timeouts, retries, circuit breaker, and deterministic fallback.
  4. Implemented decision lineage and transparent scoring.
  5. Passed synthetic data quality gate at 10k plus scale.
  6. Delivered a workflow console with phase inspection and dispatch actions.
  7. Built reproducible provisioning and verification scripts.

What we learned

  1. PM decisions require both narrative intelligence and deterministic evidence.
  2. Sufficiency criteria must be workflow-specific.
  3. Memory improves quality only when intentionally layered.
  4. Fallback strategies are mandatory for reliability.
  5. Explicit tool contracts reduce drift.
  6. Actionability changes perceived value more than analytics alone.

What’s next for SignalPM

  1. Raise benchmark pass rates with focused citation completeness and latency improvements.
  2. Fix 4 failing unit tests and stabilize async test wiring.
  3. Improve deep-mode reliability under load.
  4. Strengthen workflow dispatch observability and failure classification.
  5. Expand workspace guard and RBAC defaults.
  6. Improve section-level citation density in proposals and PRDs.
  7. Add closed-loop tracking from dispatched actions back into decision memory and priority scoring.

Built With

Share this project:

Updates