CitedOS

CitedOS shows B2B SaaS companies where they appear in Gemini, ChatGPT, and Claude answers, why competitors get cited instead, and exactly which sources to win to change that.

Comment

CitedOS — Project Story

Tagline: CitedOS shows B2B SaaS companies where they appear in ChatGPT, Gemini, and Claude answers, why competitors get cited instead, and exactly which sources to win to change that.

Inspiration

Buyers stopped asking Google first. They ask ChatGPT, Gemini, and Claude — and those engines answer with a handful of citations that decide who gets discovered and who doesn't. We watched B2B SaaS companies fly blind in this new channel: no idea whether AI engines mention them, no idea why competitors keep getting cited, and no idea what to do about it.

The existing tools bothered us too. Most show a single point-in-time "visibility score" as if it were stable — but AI citations swing 40–60% month to month. A single-run number isn't a measurement; it's a coin flip dressed up as a dashboard. We wanted to build the tool we'd trust ourselves: statistically honest, cross-engine, and ending in an actual to-do list instead of a vanity score.

What it does

CitedOS is a cross-engine AI-search visibility platform for B2B SaaS. It answers three questions:

  1. Where do you stand? It runs your buyer prompts across ChatGPT, Gemini, and Claude multiple times, then measures share of voice, mention rate, position, sentiment, and presence — every metric shown with a confidence interval and freshness timestamp, never a naked point estimate.
  2. Why do competitors win? The Source-Gap Analyzer collects every citation the engines return, scrapes the cited source pages, embeds them, and learns the citability signature — the features that separate cited pages from ignored ones for your query space.
  3. What should you do? It turns the gap analysis into a prioritized playbook: exactly which third-party sources (roughly 90–95% of AI citations are third-party, not your own site) to get into, and which on-page fixes make your pages selectable.

Around that core: a free public audit, a guided "Climb" roadmap that turns AEO into clear steps, Google Search Console and GA4 AI-referral integrations, shareable report decks, and an MCP server so your own AI agents can query get_visibility and get_source_gap directly.

How we built it

A modular monolith in TypeScript end to end: Next.js (App Router) on Vercel, tRPC as the hard authorization boundary (clients never touch the database), Prisma on Supabase Postgres with pgvector for embeddings, and Trigger.dev for the long-running collection pipelines. Billing runs through LemonSqueezy as merchant of record.

Two decisions shaped everything:

Official APIs only. We never scrape the consumer ChatGPT/Gemini/Claude apps. Each engine sits behind one collector contract with a circuit breaker: OpenAI's web-search tool, Gemini's google_search grounding (parsing groundingMetadata for real sub-queries and source chunks), and Claude's web-search url_citation annotations. The only scraping we do is of the cited source pages — public web pages — via a managed scraping API.

Statistics as a feature. Every metric is multi-run sampled. Mention rates are proportions over $n$ runs, so we report Wilson score intervals instead of raw percentages:

$$\hat{p}_{\pm} = \frac{\hat{p} + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}$$

Trend deltas only render when a two-proportion comparison is actually significant — otherwise the UI says "no reliable change" instead of inventing a story from noise.

The pipeline spine is collect → scrape → chunk → embed → extract features → aggregate, with an LLM-as-judge plus heuristic layer for mention/sentiment/position extraction, and RAG over pgvector for the source corpus.

Challenges we ran into

  • Single-run numbers lie. Our earliest prototype showed visibility jumping wildly between runs of the same prompt. The fix wasn't averaging harder — it was redesigning the product around uncertainty: multi-run sampling, confidence intervals everywhere, and honesty gates that refuse to show trends the data can't support.
  • Three engines, three citation dialects. OpenAI, Gemini, and Claude each expose citations in a completely different shape (annotations vs. grounding metadata vs. indexed spans). Normalizing them behind one contract — while preserving engine-specific detail like Gemini's query fan-out — took several rewrites.
  • Correlation ≠ causation in the gap analysis. Cited pages share features, but domain authority and brand search volume are confounders. We had to compare cited vs. not-cited sources that appeared for the same query, and split every recommendation into selectability (on-page: passage structure, fact density, schema, freshness) versus discoverability (off-site authority) — because embeddings capture the first and say nothing about the second.
  • Cost discipline. Three engines × N prompts × M runs × daily cadence gets expensive fast. We built freshness-window caching, batched embeddings, a two-tier model strategy (cheap models for the free audit, flagship for paid deep analysis), and per-plan budget guards.
  • The API surface is a proxy. The OpenAI web-search API correlates strongly with consumer ChatGPT but isn't identical. Rather than hide that, we disclose it in the product — every ChatGPT-family metric is tagged as "API surface."

Accomplishments that we're proud of

  • A production platform where honesty is the differentiator: confidence intervals, freshness stamps, and "not enough data" states instead of fake precision.
  • The full Module 2 loop working end to end — citations in, scraped and embedded sources, learned citability signature, prioritized playbook out.
  • An MCP server that makes CitedOS agent-native: any MCP client can pull visibility and source-gap data with a scoped API key.
  • 3,000+ tests, security review before anything touching customer data ships, and a pipeline that degrades gracefully — a broken collector shows stale-but-flagged data, never a blank screen.
  • Shipping the whole thing as a solo-founder-plus-AI-agents operation: spec-driven planning, agent-built implementation, human review reserved for taste and architecture.

What we learned

  • Measurement honesty is a moat. Competitors sell certainty; buyers who get burned by unstable scores come looking for confidence intervals.
  • AI search is a fan-out problem. One user prompt becomes ~8–9 engine-generated sub-queries; tracking only headline prompts misses most of the battlefield.
  • Being citable is two separate games. Discoverability gets you into the retrieval pool; selectability gets you cited from it (engines cite only ~15% of retrieved pages). Conflating them produces useless advice.
  • Your own site barely matters. With ~90–95% of citations pointing at third-party sources, AI visibility is earned-media work, not on-site SEO with new paint.
  • Building on official APIs beats scraping — compliant, stable, and rich with metadata the consumer apps never expose.

What's next for CitedOS

  • More engines: Perplexity next, then Grok, on the same collector contract.
  • From insight to execution: expanding the human-approved, draft-only agent lane that turns playbook items into ready-to-review content drafts.
  • Deeper attribution: connecting AI-referral traffic and agent-traffic analytics to revenue outcomes.
  • Defense in depth: row-level security hardening on top of the tRPC boundary as we scale to teams.
  • The long game: becoming the operating system for AI-search visibility — measurement, diagnosis, and execution in one loop, available to both humans and their agents.

Built With

  • anthropic-claude-api
  • d3.js
  • gemini-api
  • google-search-console-api
  • lemonsqueezy
  • model-context-protocol
  • next-auth
  • next.js
  • node.js
  • openai-api
  • pgvector
  • postgresql
  • prisma
  • react
  • resend
  • scrapingbee
  • supabase
  • tailwindcss
  • trigger.dev
  • trpc
  • typescript
  • vercel
  • vitest
  • zod
Share this project:

Updates