-
-
From contract change to DataHub impact, Codex repair, evaluation, Draft PR, and verified writeback.
-
Repository tests pass, but MCPatch rejects the candidate because cents were incorrectly interpreted as dollars.
-
Across 54 trials, MCPatch delivered 9/9 discovery, 9/9 ready-for-review outcomes, and zero unnecessary changes.
-
A real Codex attempt produces a minimal repair, independently evaluated and published as a human-reviewed Draft PR.
-
DataHub maps agents, skills, tools, repositories, owners, lineage, and governance into one repair context graph.
-
MCPatch separates affected consumers from repair targets, and proves which repositories must remain unchanged.
-
Checksum-backed evidence confirms the migration record was written to DataHub and matched on verified readback.
-
Seven independent layers verify policy, tests, contracts, semantics, governance, integration, and unchanged controls.
About the project
MCPatch is a DataHub-powered coding agent that helps teams safely update AI agents when a shared tool, API, skill, or schema changes. It uses organizational context to find every affected consumer, identify the correct repositories to modify, generate bounded repairs, validate them independently, and open draft pull requests for human review.
Inspiration
AI agents increasingly depend on shared tools, APIs, skills, and data contracts. A small contract change can silently break agents across several repositories.
Finding those failures is harder than searching for an old function name. The real dependencies may be indirect. One repository may own the shared skill, another may contain the affected agent, and a third may be a protected control that must not change. Important information such as lineage, ownership, business meaning, and governance rules usually lives outside the source code.
I built MCPatch because this is exactly the kind of problem where an agent needs organizational context, not just a larger prompt. DataHub provides that context through its graph.
What it does
MCPatch turns a contract change into a safe, reviewable repair workflow:
- It compares the old and proposed contract.
- It reads lineage, ownership, repository mappings, glossary meaning, and governance metadata from DataHub.
- It identifies affected AI agents, including indirect consumers.
- It separates repositories that require a repair from repositories that must remain unchanged.
- It runs a coding agent inside tightly restricted repository, command, path, and attempt boundaries.
- It sends each candidate to an independent evaluator that checks policy, contracts, semantics, governance, integration behavior, and unchanged controls.
- It opens draft pull requests for accepted repairs.
- It writes the verified migration result back to DataHub and reads it back before reporting success.
MCPatch never merges or deploys generated code. Humans remain responsible for the final decision.
How we built it
I built MCPatch in Python using DataHub OSS and the Agent Context Kit as the context layer.
The project represents agents, shared skills, tools, repositories, owners, contracts, glossary terms, lineage, and governance controls in DataHub. MCPatch performs primary reads from the graph and fails closed when required context is missing or inconsistent. It does not silently replace missing DataHub information with a local guess.
A deterministic contract diff starts the workflow. The resulting DataHub context is converted into a typed impact plan that distinguishes three important concepts:
- affected consumers;
- repositories that require changes;
- repositories that must remain unchanged.
Codex generates candidate repairs inside isolated Docker environments. A separate evaluator receives hidden acceptance checks and verifies each candidate independently. Accepted changes can be published as GitHub draft pull requests, while the final migration record is written back to DataHub as reusable context for future people and agents.
I also built a credential-free replay viewer and static GitHub Pages deployment so judges can inspect the workflow, generated patches, evaluations, DataHub writeback evidence, and benchmark results without configuring external services.
The stack includes Python, DataHub OSS, the Agent Context Kit, Codex CLI, Docker, GitHub, Pydantic, JSON Schema, Typer, Starlette, and deterministic evidence generation.
Challenges we ran into
The hardest problem was distinguishing an affected agent from the repository where a repair belongs. A shared skill can affect several agents, but the correct fix may belong in only one repository. Treating every affected consumer as a repair target creates unnecessary and potentially dangerous changes.
Another challenge was making generated code observable and safe. A patch can compile and pass repository tests while still violating business meaning, governance requirements, or cross-repository integration behavior. I addressed this by separating candidate generation from evaluation and by checking repositories that are expected to remain unchanged.
Reproducibility also required more work than expected. Provider output is naturally variable, while benchmark evidence must remain verifiable across machines and Python versions. I built typed artifacts, canonical serialization, checksum-bound manifests, frozen inputs, deterministic replay, and exact validation gates. I also fixed platform-specific issues involving floating-point summaries and Docker bind-mount cleanup instead of weakening the verification rules.
Finally, I had to preserve unsuccessful outcomes honestly. Model failures and policy rejections remain in the evidence rather than being retried until a favorable result appears.
Accomplishments that we're proud of
The biggest accomplishment is that MCPatch works as a complete loop. It reads meaningful context from DataHub, produces real code changes, evaluates those changes independently, opens draft pull requests, and writes verified results back to the graph.
The repository includes public sample outputs from real Codex runs, five draft pull requests, independent evaluation reports, DataHub migration records, and readback receipts.
I also completed a paired 54-trial benchmark across billing, risk, and shipment scenarios:
| Task | Baseline | MCPatch |
|---|---|---|
| Exact impact discovery | 0/9 | 9/9 |
| Repair completed within budget | 9/9 | 8/9 |
| End-to-end result ready for review | 0/9 | 9/9 |
The agent-only workflow made six unnecessary repository changes and missed six required repair targets. MCPatch made zero unnecessary changes and missed zero targets.
The repair-only baseline matched or beat MCPatch in all nine pairs, including one MCPatch policy rejection. I kept that result because trustworthy evaluation matters more than presenting a perfect score.
What we learned
The most important lesson was that context quality matters more than prompt size. Source code can show that a symbol exists, but it cannot reliably explain who owns it, which agents depend on it indirectly, what business meaning must be preserved, or which repositories must not change.
I also learned that false positives are not harmless in an automated repair system. Every unnecessary repair target increases cost, review effort, and risk. DataHub's graph makes it possible to reason about the full fleet before generating code.
Independent evaluation is equally important. The same agent that wrote a patch should not be the only system deciding whether the patch is safe. Separating generation from evaluation made failures visible and prevented plausible-looking code from being treated as verified work.
Finally, writing results back to DataHub changes the workflow from a one-time agent response into durable organizational knowledge. Future people and agents can discover what changed, what was repaired, what was verified, and what still requires review.
What's next for MCPatch
The next step is to test MCPatch with larger repositories, more contract formats, additional programming languages, and real platform-team workflows.
I also want to:
- package the impact-analysis and migration-writeback workflow as reusable DataHub skills;
- support additional repository and CI providers;
- add cost and latency observability;
- improve reviewer feedback loops;
- expand evaluation for organization-specific policies;
- test production-scale lineage graphs;
- contribute reusable documentation, fixes, and integration patterns back to the DataHub open-source ecosystem.
The long-term goal is not to remove human review. It is to give reviewers a complete, evidence-backed repair package so they can make safer decisions much faster.
Log in or sign up for Devpost to join the conversation.