ingestion.work

Inspiration

Every company has the answers it needs. They're just scattered. Policies live in one wiki, decisions in meeting notes, fixes in support tickets, and the numbers sit in databases that only the data team can query. A simple question like "How bad is the Android crash rate since our last release, and is a fix planned?" turns into a Slack hunt for the docs and a ticket for the data, and the answer arrives days later.

We were inspired by the shift to agentic analytics: instead of question → ticket → dashboard, an AI agent explores the data, writes the SQL, checks its own work and answers in seconds. We wanted one brain that handles both halves of company knowledge, documents and data, and that you can actually trust: every claim cited, every query auditable, every permission enforced.

What it does

Ingestion is a Company Brain. Ask in plain language and it:

• searches documentation, meeting notes, FAQs, policies, product and support content, and internal tools;
• walks a knowledge graph of teams, services, people and incidents, plus a real org chart, for ownership and impact questions;
• delegates data questions to a data-analyst sub-agent that explores ClickHouse, checks real column values, writes SQL, sanity-checks results and draws charts;
• returns one answer where every sentence cites either a document passage or the exact SQL query behind it.

How we built it

• Agents: a main agent on Claude Opus plans, searches, delegates and cites. A data-analyst sub-agent on Claude Sonnet does the SQL work. Both share one streaming tool-use loop with full tracing.
• Retrieval: heading-aware chunking with BM25 ranking by default, and a switchable HydraDB backend for hybrid semantic search. BM25 scores a chunk $D$ for a query $Q$ as

$$ \text{score}(D,Q)=\sum_{t\in Q}\text{IDF}(t)\cdot\frac{f(t,D)\,(k_1+1)}{f(t,D)+k_1\left(1-b+b\,\frac{|D|}{\text{avgdl}}\right)} $$

with $k_1=1.2$ and $b=0.75$. Superseded documents get a $0.8\times$ penalty, so the current policy wins.

• Analytics: ClickHouse Cloud holds about 38.4M product events plus billing, support and HR tables. Because storage is columnar and sorted by (event_name, date, workspace), a full scan of 38.4M rows takes about 70 ms locally, so the agent can afford to explore and double-check.
• Context protection: query results stay server-side. The model only sees a 30-row preview and a handle like q3, and charts reference that handle. A 5,000-row result costs the model about $\frac{30}{5000}=0.6\%$ of its raw size in context.
• Human in the loop: before running a query, the analyst asks ClickHouse for a row estimate (EXPLAIN ESTIMATE). If $\widehat{\text{rows}} > 3\times10^{7}$, the answer pauses for an Approve/Deny decision.
• Real HR data: we connected a real HR system (Ascend, on Postgres/Neon) through a read-only, column-restricted role. Every person gets a stable pseudonym, and the role physically cannot read emails, credentials or any free text. A leak scan found 0 real names in the imported data.
• Permissions in the database: each persona queries as its own ClickHouse user with grants, so HR ratings are unreadable to non-HR personas even if the model were tricked.
• Product surface: shareable reports with a full audit trail, scheduled digests, Langfuse-style traces stored in ClickHouse, an MCP server so Claude Code can use the brain directly, and deployment on Vercel at ingestion.work.

What we learned

• Grounding beats eloquence. Users trust an answer they can click through to the source, so citations had to cover numbers (SQL) as well as text.
• Data is messy, and agents must check. Our seed data had a legacy event name (Dashboard Viewed vs dashboard_view) and legacy country spellings. Telling the analyst to inspect real values before filtering made answers noticeably more accurate.
• Permissions belong in the database, not the prompt. Prompts can be argued with; ClickHouse grants and Postgres column privileges can't.
• Measure everything. Our eval asks 10 cross-source questions as different personas and checks for the required facts and citations. It scored 10/10 with both search backends, averaging $\approx 15\,\text{s}$ and $\approx \$0.05$ per question:

$$ \bar{c}=\frac{\$0.51}{10}\approx\$0.05,\qquad \bar{t}=\frac{1}{10}\sum_{i=1}^{10}t_i\approx 15.0\,\text{s} $$

Challenges we faced

• Privacy with real people's data. We excluded all free text, pseudonymized names (including names embedded inside team names and goal titles), and applied a small-group rule: no averages for groups with $n<5$.
• Serverless vs. stateful agents. On Vercel, the "Approve" click can land on a different instance than the one running the agent. We moved all state (approvals, reports, schedules, memory) into ClickHouse so any instance can resume any conversation.
• Deploy surprises. Vercel's file tracer tried to bundle our 760 MB local ClickHouse binary, and TypeScript .ts imports broke after compilation. We fixed both with explicit excludes and rewriteRelativeImportExtensions.
• Undocumented edges. HydraDB rejected a metadata field named source_type, which is reserved, with a generic error. It also reported sources as searchable before their chunks were stored. We bisected the first with throwaway databases and made our sync wait for real chunk availability.
• Agents that loop. An early org question made the agent guess names until it ran out of turns. An "overview" mode for the org tool and a forced final answer cut that run from about 20 failed lookups to 2 calls in 7 seconds, for about \$0.03.

What's next

Live connectors (Slack, Jira, Drive, and real-time Postgres → ClickHouse sync), per-user SSO with roles instead of shared passwords, and safely redacted free text so HR can ask about themes in feedback too.

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for ingestion.work

Built With

• claude
• hydradb
• neon
• vercel