SourceFit

SourceFit is a DataHub-powered agent that measures a vendor dataset’s real lift, blocks usage-rights violations, generates the integration, and writes the decision back to your data catalog.

Comment

Inspiration

Teams buy external data on promises.

A vendor can show a polished sample and an impressive metric without answering the questions that matter inside a real company:

  • Does this dataset improve the production model?
  • Does it add information we do not already own?
  • Are we legally allowed to use it for training?
  • Is the improvement worth the price?
  • Who will build and validate the integration?

We built SourceFit because a procurement decision should use the same context that data engineers, ML teams, and governance teams rely on. It should also leave behind more than a score.

What it does

SourceFit is a DataHub-powered procurement agent.

It reads the production model's schema and dataset lineage from DataHub through MCP. This gives the agent a trusted comparison set: the fields and datasets already feeding the model.

SourceFit then:

  1. Measures baseline-versus-augmented model lift for every vendor.
  2. Checks usage rights before producing a procurement score.
  3. Measures redundancy against the data the company already owns.
  4. Considers price and integration effort.
  5. Selects the best eligible vendor.
  6. Generates and validates the production integration.
  7. Writes a checksummed Procurement Passport back to DataHub.
  8. Reads the Passport back and accepts the write only when the digests match.

The decision that matters

Our demo evaluates three vendors using the same model, metric, and policy.

Vendor Decision Reason
Vendor A Rejected, 32.21 Too much overlap for the value delivered
Vendor B Blocked, no score Best raw lift, but production training is forbidden
Vendor C Recommended, 69.93 Useful lift, permitted use, acceptable cost and effort

Vendor B produces the highest raw lift: 11.1%.

SourceFit still refuses to rank it.

Its license forbids production training, so the rights gate removes it before a procurement score can exist. Vendor C wins because it provides useful, permitted, net-new signal.

Governance happens before scoring, not as a warning after the decision.

How we built it

DataHub MCP is the agent's context layer. It supplies the production schema and lineage that make the vendor comparison meaningful.

The evaluation runs asynchronously as a Cloud Run Job. A Cloud Run service exposes the API, Firestore stores durable run state, Cloud Storage preserves immutable evidence, and Firebase App Hosting serves the judge-facing experience.

The evaluation and modeling layer uses:

  • Python
  • pandas
  • scikit-learn
  • DuckDB
  • Paired baseline and augmented model tests

The public application uses:

  • Next.js
  • React
  • TypeScript
  • Firebase App Hosting

For the winning vendor, SourceFit generates 17 artifacts across Python, SQL, dbt, configuration, policy, review, and DataHub metadata. It then executes 35 validators before the integration can be marked ready.

How we use DataHub

DataHub is not a decorative metadata lookup in SourceFit.

The agent uses DataHub context to determine:

  • Which fields already feed the production model
  • Which vendor features are redundant
  • How datasets connect to the model through lineage
  • Which asset should receive the final procurement decision
  • Whether the written Procurement Passport can be verified after readback

Without DataHub context, Vendor B appears to be the obvious winner.

With DataHub, the agent discovers that Vendor A is mostly redundant, Vendor B has a rights conflict, and Vendor C is the eligible net-new signal.

The final decision is written back to DataHub so the next person or agent inherits the result instead of repeating the evaluation.

Challenges we ran into

The hardest problem was teaching the agent to refuse an attractive answer.

Vendor B genuinely has the best lift. A score-first system would rank it first and hide the license conflict inside a penalty or footnote. We implemented usage eligibility as a hard gate before scoring.

The second challenge was evidence. A polished dashboard is easy to stage. We preserved authorized DataHub read and write proof, checksummed generated artifacts, explicit capability skips, and immutable run state. Every important claim in the public experience connects to a test, receipt, or generated file.

We also separated browser request latency from durable ML work. The evaluation runs as an asynchronous Cloud Run Job instead of pretending that model training and artifact generation fit inside a short web request.

Accomplishments that we are proud of

  • A real DataHub read changes which vendor features count as redundant.
  • The highest-lift vendor is visibly blocked instead of quietly down-ranked.
  • The winner produces 17 inspectable integration artifacts.
  • The generated integration passes 35 validators.
  • The Procurement Passport is verified after DataHub readback.
  • The public six-view experience requires no login.
  • The repository includes the source, setup instructions, evidence, and sample outputs.

What we learned

Context is what separates a useful agent from a generic ranking script.

Lift alone is not a purchasing decision. Governance is not a post-processing label. Code generation is only valuable when its inputs come from the real schema and its output is validated.

We also learned that refusal can be one of the strongest trust signals an agent provides. Showing why SourceFit will not score Vendor B says more than another green dashboard card.

What's next

The next step is a guided evaluation flow where a team selects a DataHub asset, maps an internal sample to vendor samples, records commercial and usage terms, and starts an isolated evaluation.

We would also add:

  • Warehouse-specific integration targets
  • Richer policy review
  • Human approval before procurement actions
  • Additional model families and evaluation metrics
  • Queue warm-up for faster on-demand runs
  • Continuous monitoring before vendor renewal

Demo boundaries

The public experience shows one preserved, deterministic fixture run so every judge sees the same decision and evidence.

  • DataHub authorized integration: real
  • DataHub read and write proof: real
  • Evaluation execution: real
  • Cloud publication: live
  • Generated artifacts: real and inspectable
  • Evaluation input: deterministic fixture
  • Arbitrary customer uploads: not included in the public hackathon demo

Loading the website does not perform a new DataHub mutation. It displays preserved evidence from authorized operations that already completed.

Built With

Share this project:

Updates