FedFathom

What it does

Federal procurement data is public, but understanding it often requires knowing the right agency names, acquisition terminology, NAICS and PSC codes, identifiers, and filters. FedFathom was inspired by a simple idea: contractors should be able to describe what they do in their own words and receive a clear, source-grounded view of their federal market.

A user can enter a capability such as:

“I maintain HVAC systems for hospitals and large facilities.”

FedFathom searches historical federal award records and surfaces:

Similar awards and representative contracts
Agencies and offices buying related services
Recent incumbent and winning vendors
Relevant NAICS and PSC codes
Procurement language used in actual awards
Contract values, locations, and competition details
Period-of-performance and possible recompete signals
Links to source records for independent verification

Users can filter by date, agency, vendor, NAICS, PSC, and location. Results are grouped around the underlying award so modifications and related transactions do not overwhelm the research experience.

FedFathom uses an explainable heuristic ranking rather than presenting an opaque prediction:

$$ S(q,d)=w_tS_{\text{text}}+w_cS_{\text{codes}}+w_bS_{\text{buyer}}+w_lS_{\text{location}}+w_rS_{\text{recency}} $$

Each result remains connected to evidence from the source dataset.

The staging product also supports Google and Microsoft account creation, five introductory searches per account, and Stripe-backed subscription access after the introductory allowance.

How we built it

FedFathom grew out of an entity-resolution project for matching messy customer names. That first system normalized names, generated MinHash signatures, used locality-sensitive hashing (LSH) to retrieve likely candidates, and scored matches using weighted Jaccard-style signals. Originally, I loaded the dataset into memory for lightning quick retrieval, but this solution isn't scalable.

We realized the same indexing and matching principles could help contractors navigate inconsistent federal award records. We generalized the original index lifecycle into a reusable dataset-search architecture and made Federal Discovery its first major application.

The application uses:

React, Vite, and TypeScript for the research workbench
Node.js, Express, and TypeScript for the API
PostgreSQL 16 as the durable source of truth
Docker Compose for repeatable local and staging environments
Nginx and Caddy for static web delivery, reverse proxying, and HTTPS
AWS Lightsail for the full-scale staging environment

Federal award transactions are streamed into PostgreSQL rather than loaded into one enormous in-memory structure. Index construction is batched, resumable, checkpointed, and organized into independently staged segments. Completed segment sets can be activated atomically without replacing a working index in place.

Search begins with bounded exact-value, curated-term, and text-signature lookups. FedFathom then hydrates a capped candidate set and applies dataset-specific ranking, summaries, facets, and source attribution.

Each admitted search creates a server-owned Search Execution containing a stable result snapshot. Paging, sorting, refreshing, or returning from Stripe reuses that execution instead of consuming another introductory search or rerunning expensive retrieval.

Authentication and billing are enforced by the backend using Google and Microsoft OIDC, opaque PostgreSQL sessions, session-bound CSRF protection, exact-origin CORS, replay-safe Stripe webhooks, local entitlements, rate limits, and separate customer and operator permissions.

Challenges we ran into

The first major challenge was scale. The staging corpus grew to more than 36 million award transactions. An early dense projection design would have required too much storage, so we redesigned retrieval around storage-bounded exact signals, curated procurement terms, and fixed-fanout signatures.

We also learned that LSH is not automatically fast just because candidate lookup is theoretically efficient. Bucket fan-out, segment count, database hydration, scoring, and response shaping must all be explicitly bounded.

Stable pagination introduced another difficult problem. If the active index changes while a user moves through results, later pages can contain duplicates, omit awards, or reorder unexpectedly. We addressed this with immutable Search Executions tied to bounded index snapshots and opaque, account-bound continuation tokens.

Concurrency exposed a production-blocking PostgreSQL issue: statements intended to be part of one transaction could run through different pooled connections. We corrected the repository architecture so BEGIN, every enclosed statement, COMMIT or ROLLBACK, and client release all use one checked-out PostgreSQL session.

Security was another broad challenge. We needed to protect expensive search operations from unauthenticated calls, client-forged roles, duplicate submissions, webhook replays, and races between introductory usage and subscription activation. Admission, idempotency, entitlement selection, rate limiting, and usage consumption are now serialized before expensive work begins.

Finally, staging forced us to solve operational problems that prototypes can avoid: TLS, private secret delivery, provider callbacks, multi-hour jobs, disk capacity, reboot recovery, cost controls, and non-destructive rollback.

Accomplishments that we're proud of

We are proud that FedFathom evolved from a name-matching prototype into a complete, source-grounded federal market research platform.

The staging database contains 36,071,263 award transactions covering FY2021 through FY2026 year-to-date. We completed a bounded 405,410-document staging pilot to measure index construction, storage consumption, and recovery behavior before authorizing the full build.

Other accomplishments include:

Natural-language federal market research grounded in historical awards
Explainable ranking with source-linked evidence
Stable, account-bound search executions and pagination
Five non-recurring introductory searches per account
Secure Google and Microsoft authentication
Stripe Checkout, subscription projection, and customer portal support
Resumable imports and segmented index construction
Atomic index activation that preserves the last working generation
A responsive and accessible workbench tested down to narrow mobile widths
Public pricing, data-source, privacy, terms, support, cancellation, and account-deletion information
Hundreds of automated API, frontend, database, architecture, and security checks
A production-built HTTPS staging environment at staging.fedfathom.com

We are especially proud that the system remains deterministic and inspectable. FedFathom helps users interpret evidence without hiding retrieval behind an opaque answer.

What we learned

We learned that public data is not necessarily accessible intelligence. The difficult part is connecting inconsistent records and translating them into questions a business actually asks: Who buys this? Who wins? Which codes matter? What language should I watch? What might be recompeted?

We learned that explainability is a product feature, not just a technical detail. A useful result needs both a reason and a path back to the source.

At scale, boundedness matters more than an algorithm’s label. Every candidate pool, segment fan-out, database query, result page, and expensive operation needs an explicit limit.

We also learned that database transactions must be verified against real concurrency behavior. A unit test using a convenient pool mock cannot prove that every statement uses the same backend session.

Finally, we learned that deployment is part of the product. Storage headroom, secret handling, recovery, provider configuration, and rollback determine whether impressive code can become a trustworthy service.

What's next for FedFathom

Our immediate milestone is completing readiness-gated activation and release verification for the full staging index.

Next, we plan to introduce a Recent Awards mode backed by incremental SAM.gov synchronization. New and updated records will become resumable “hot” index segments while reusing the existing ranking, paging, metering, authorization, and source-evidence foundations.

We also want to expand into federal opportunities and grant datasets, improve ranking through representative user feedback, and provide richer trend analysis across buyers, vendors, codes, locations, and time periods.

The long-term vision remains simple: