Sybil: Customer Observability for B2B SaaS

Sybil correlates your product's telemetry to the revenue each customer represents and surfaces the accounts silently breaking — ranked by revenue at risk — before they churn.

Comment

Inspiration

Customer Success teams have signals for churn risk (e.g., health scores, usage trends, support tickets) but they're mostly lagging or self-reported. A support ticket is the clearest example: by the time it's filed, the customer's already unhappy. And most never file one: ThinkJar's Esteban Kolsky found that only 1 in 26 unhappy customers ever complain — the other 25 just churn.

But a leading, real-time signal is sitting in their own product telemetry; it just isn't connected to revenue.

What Sybil does

Sybil connects a telemetry failure to the revenue behind it, and scores each account by renewal dollars at risk and the type of failure. It notifies a Customer Success Manager via Slack when a real service issue has been detected, and helps the CSM immediately send a proactive message to help mitigate churn risk and maintain customer trust. When the issue has been resolved, Sybil also lets the CSM close the loop with the customer.

How I built it

I chose Aurora PostgreSQL because the detection is a SQL query. It uses per-account z-score baselining plus exposure scoring blended into one revenue-weighted ranking, running on an engine that scales to zero between incidents to minimize costs.

I built the demo around an identity/SSO vendor, because there a single failure is both an outage and a renewal risk: a terminated employee's de-provisioning webhook silently fails, their admin session stays live, and nobody notices until the security review or the renewal call.

The domain choice seemed like a natural fit since identity and SSO telemetry is likely to be tenant scoped, so attributing a failure to a customer account is a join, not a guess.

Sybil is not just an alert on any telemetry error - different failures need to be scored differently. In this demo, we have:

  • High-volume sync failures → a rate anomaly. I baseline each tenant against its own 7-day hourly history and flag a live burst that blows past mu + 3σ. A z-score works here because there's a baseline to deviate from.
  • A single terminated-admin-still-logged-in event → a z-score is useless, because the baseline is zero. One occurrence is already a P1. So those get exposure scoring: blast radius × sensitivity × dwell time.

Both blend into one revenue-weighted risk score, so a whale account with a confirmed exposure near renewal outranks a bigger account that's merely showing some instability. And it's all SQL. This is what the CX research keeps landing on: two correlated signals predict churn far better than either one alone. The difference here is that Sybil's signals resolves in real-time to a dollar figure, proximity to renewal, and a named owner, and give CSMs the ability to reach out to a key at-risk account almost immediately.

Challenges I ran into

The RDS Data API can't bind native Postgres enums. I had a clean pgEnum schema; the Data API path rejected it. I moved the whole schema to text columns + CHECK constraints to keep the same safety without the binding.

My anomaly detector quietly stopped firing for some accounts, and it took me a while to see why. I cache the 7-day baseline in a materialized view so the live query stays fast — but if that rollup ever refreshed while an incident was live, the burst got baked into the baseline itself. The tenant's "normal" rate jumped, so a real spike no longer looked abnormal and the account never surfaced. The fix was to recompute the rollup on reset, so a cleared incident can't keep poisoning the baseline.

Accomplishments that I'm proud of

The detection is entirely SQL — no ML black box. That means a CSM can trust the flag is rooted in real, inspectable data: this account's sync failures are many times its own baseline, with a confirmed exposure — not an opaque score.

I got the real provider pipeline to drive the whole UI. A raw Sentry webhook hits the same ingest endpoint a production integration would; detection picks it up, the board goes red, an incident opens, and the owner gets pinged in Slack.

And it stays cheap and closed: the cluster scales to zero between incidents, and the app is designed to reach the database over the RDS Data API — no open port.

What I learned

Use the right detector for the signal. I started trying to z-score everything, but a rare event — one terminated admin still holding a live session — has a baseline of zero, so a z-score is meaningless. Different failures genuinely need different math.

The bigger realization came later: those detectors aren't identity-specific — they're statistical patterns. A sync-failure burst here is the same rate-anomaly detector as an API-error burst at a payments company; only the normalized event model underneath changes. That reframed what the product actually is. The hard part isn't writing custom SQL per customer — it's knowing which detector fits which signal. That part generalizes.

What's next for Sybil: Customer Observability for B2B SaaS

  • Real ARR and Renewal Dates. The dollar weighting is seeded for the demo; in production it would upsert from the system of record (CRM) through that same ingest path, so every number traces back to billing.
  • General telemetry attribution. The hardest part will be normalizing how to extract a customer identifier out of telemetry data from the providers - whether it's a tag, structured field data, or something else — and how to handle when no identifier exists.
  • A standardized library of detectors. Because everything lands in one normalized model, the detectors are reusable primitives — rate anomaly, rare-event exposure, latency degradation, policy violations — not bespoke SQL per customer. The IP is the library and knowing which detector fits which signal.
  • Onboarding as configuration, not a rebuild. New customers map their event types to detectors and tune the risk weights (how much ARR vs. exposure vs. renewal proximity matters to them); per-tenant baselines are learned automatically. A standard core with an escape hatch for the long tail — a product, not a consulting engagement.
  • Close the loop on outcomes. Track whether proactive outreach actually reduced churn, so the risk weights can learn from what worked — the difference between measuring experience and steering it.
  • Real auth and multi-CSM workflows, replacing the demo SSO gate.

Built With

Share this project:

Updates