Histograph

Lineage-aware observability and incident response for production ML.

Comment

Histograph

Inspiration

Production machine-learning failures rarely look like ordinary application failures. An API can remain healthy while a new model threshold quietly reduces fraud recall, or a feature release changes the distribution of values reaching the model.

Existing monitoring can show that a metric changed, but the response is often fragmented across dashboards, metadata systems, deployment tools, and incident channels. Engineers still need to answer:

  • What changed before the regression?
  • Is the affected asset connected to this model?
  • Which release should be stopped?
  • Who must approve that action?
  • Did the rollback genuinely restore performance?

We built Histograph to connect those steps into one evidence-driven incident-response loop.

What it does

Histograph is a production ML incident-response platform. It receives model predictions, delayed outcomes, deployment observations, and upstream change events. Continuous monitor workers evaluate that evidence and open incidents when deterministic thresholds are crossed.

For our reference demonstration, we serve a mobile-money fraud model with two releases:

  • v1: The stable release using a threshold selected from validation data.
  • v2: A 10% canary with an intentionally incorrect decision threshold of 0.99.

The canary monitor compares both releases using held-out transactions from the same time window. Recall is calculated as:

$$ \text{Recall} = \frac{TP}{TP + FN} $$

The release difference is measured as:

$$ \Delta_{\mathrm{recall}} = \mathrm{Recall}{v2} - \mathrm{Recall}{v1} $$

When v2 loses enough recall, Histograph opens an incident and investigates the registered model through DataHub. It reads the model entity, upstream features and datasets, downstream relationships, and ownership through the DataHub MCP server.

If the evidence supports a probable cause, Histograph proposes a protective action. For a GitOps deployment, it opens a real rollback pull request that changes candidate traffic from 10% to 0%. The engineer merges the PR, making the approval explicit and auditable.

A reference reconciler applies the merged manifest to the serving runtime. Histograph then sends fresh held-out traffic through the recovered deployment and waits for delayed outcomes. The incident is resolved only after new evidence verifies that performance recovered.

The complete flow is:

Predictions and delayed outcomes
        ↓
Continuous performance monitor
        ↓
Incident and DataHub investigation
        ↓
Probable model-release cause
        ↓
Rollback pull request
        ↓
Engineer merges the PR
        ↓
Runtime removes candidate traffic
        ↓
Fresh predictions and outcomes
        ↓
Recovery verification
        ↓
Incident resolution and DataHub write-back

How we built it

Histograph separates high-volume observations from durable control-plane state:

  • FastAPI provides event ingestion, deployment, monitor, incident, action, and playground APIs.
  • ClickHouse stores prediction and outcome telemetry.
  • PostgreSQL stores deployments, monitors, incidents, investigations, approvals, actions, timelines, and scenario state.
  • Continuous workers use database leases and FOR UPDATE SKIP LOCKED to prevent duplicate monitor evaluation or action execution.
  • DataHub OSS and its MCP server provide model, feature, dataset, lineage, and ownership context.
  • A GitHub App imports .histograph deployment manifests, opens rollback PRs, and processes signed merge and deployment events.
  • The reference runtime loads the trained model, routes deterministic 90/10 traffic, accepts delayed outcomes, and emits runtime evidence through a durable outbox.
  • The reference reconciler applies the exact Git revision approved through GitHub and reports deployment status.
  • Next.js and Vercel power the public dashboard and controlled demonstration at app.histograph.ai.

The deployment manifest connects the model to its DataHub URN, input and output schemas, examples, stable and candidate releases, feature configuration, and rollback targets. Histograph imports these resources from the same Git revision so the runtime contract cannot silently mix files from different releases.

For the model environment, we used the versioned Synthetic Mobile Money Transaction Dataset. DuckDB converts the source CSV into compressed Parquet, and the training pipeline compares logistic regression with histogram gradient boosting.

We use a chronological split instead of a random split:

55% training
→ 24-hour label gap
→ validation through 75%
→ 24-hour label gap
→ held-out test period

The controlled scenario replays 1,000 held-out transactions through the real runtime, records predictions and delayed outcomes, waits for the continuous monitor, and follows the same investigation and remediation path used by the control plane.

How DataHub changes the decision

We did not want DataHub to be a decorative lookup or another place to store telemetry. ClickHouse remains responsible for prediction data, while DataHub supplies organizational context that telemetry alone cannot provide.

Without DataHub, Histograph can prove that performance degraded, but an upstream feature incident cannot safely determine which change is connected to the affected model. The correct behavior is to abstain and request more evidence.

With DataHub, Histograph constrains the investigation to assets in the registered model’s lineage and proposes a bounded action against the matching release. We added an enabled-versus-disabled regression test to prove that DataHub changes the operational recommendation instead of merely adding text to an incident report.

Challenges we faced

Separating correlation from confirmed causality

A metric regression occurring after a deployment is evidence, but it is not complete proof. We introduced explicit evidence stages:

  1. Detected: A deterministic monitor crossed its threshold.
  2. Probable cause: The same-window candidate degraded and the runtime confirmed it was actively serving.
  3. Confirmed cause: The implicated release was removed and a fresh recovery window passed.

This prevents Histograph from overstating what the available evidence proves.

Reconstructing deployment state at the incident time

The canary could have started before the narrow investigation window while still serving during the incident. Widening the time window would introduce unrelated changes.

We solved this by reconstructing the candidate’s runtime-confirmed state as of the incident timestamp. That allowed Histograph to identify the active 10% canary without accidentally using a later deployment observation.

Handling delayed outcomes correctly

Fraud outcomes do not necessarily arrive with predictions. We had to correlate each outcome with its original prediction and ensure monitor windows contained enough labeled samples.

Recovery verification was even stricter. Pre-remediation observations cannot prove recovery, so Histograph requires prediction and outcome pairs created strictly after the rollback.

Detecting constant-distribution feature shifts

A normal histogram can collapse when every baseline value is identical. Our first PSI implementation could report zero even when a constant feature moved to a completely different value.

We corrected the bin handling and added a focused regression test so constant-value shifts are detected rather than disappearing mathematically.

Keeping desired state separate from observed state

A merged GitHub PR proves that an engineer approved a desired configuration. It does not prove that the runtime applied it.

Histograph therefore tracks GitHub state, deployment status, runtime observations, and model health separately. A successful adapter or merged PR is not enough to resolve an incident.

Running a realistic public system

The demonstration includes a self-hosted DataHub stack, PostgreSQL, ClickHouse, the Histograph API and workers, the model runtime, and the reconciler.

We kept these services on a VPS while deploying the Next.js client separately on Vercel. This reduced load and isolated the public interface from the data plane.

What we learned

The most important lesson was that monitoring and incident response are different problems. A metric tells us that something is wrong. Responding safely requires lineage, ownership, deployment history, approval, runtime confirmation, and fresh recovery evidence.

We also learned that:

  • Lineage is valuable when it changes or constrains an operational decision.
  • A merged rollback PR is approval, not proof of execution or recovery.
  • Delayed labels must be treated as a first-class part of model monitoring.
  • Desired Git state and observed runtime state should never be conflated.
  • An investigation agent should abstain when evidence is insufficient.
  • Incident resolution should mean recovery was verified.
  • Manual closure should require an engineer and a recorded reason.
  • A credible demonstration needs real traffic, outcomes, infrastructure, lineage, and approval.

Dataset credit

The reference environment uses:

Azamuke, Denish (2024), Synthetic Mobile Money Transaction Dataset, Mendeley Data, Version 2.

The repository does not redistribute the source dataset. Its preparation pipeline preserves the source version, citation, and file checksums.

Built With

Share this project:

Updates

Submission history