DataForge

• 

  Screenshot of DataForge's human review interface showing three flagged fields with source-text highlighting and reasoning

Inspiration

The main objective is to double the number of doctors or health experts, by introducing virtual AI doctors that are backed by Realtime data and medical history of a patient, hence giving them access to doctors or experts 24/7

Every team that wants to build or fine-tune their own AI model runs into the same wall: the model is only as good as the data you feed it, and getting well-organized, annotated data is brutally slow. Someone has to sit down and manually label documents, and if you're working in a specialized field, you need an actual expert to do it — not just anyone with time on their hands.

We started asking: what if instead of writing a labeler for radiology reports, and then a completely different one for legal contracts, and another for customer support tickets, you built ONE system that could be pointed at any kind of document just by handing it a config file? That question turned into DataForge.

We landed on medical data specifically because it's one of the clearest, highest-stakes examples of this problem. Existing tools like CheXpert and NegBio can pull structured labels out of chest X-ray reports, but they're locked to a fixed set of about 14 diagnoses. The moment you need a different field extracted, or you're working with a completely different kind of report, those tools can't help you. We wanted to build something that could.

What it does

DataForge takes any raw, unstructured document — a radiology report, a pathology report, and eventually anything else — and turns it into clean, structured, fine-tuning-ready training data, using a small config file called a Domain Pack to tell it what to look for.

A team of five AI agents does the actual work:

• A Parser splits the raw text into its logical sections.
• A Structuring Agent (powered by Google's LangExtract library) pulls out the fields we care about, and tags every single one with the exact sentence it came from — so nothing is a black box.
• An Alignment Agent cross-checks extracted findings against an attached image, when there is one.
• A Verifier Agent independently re-reads the source text and checks whether what got extracted is actually supported by it, or whether it's an inference that snuck in.
• Based on how confident the Verifier is, the record either gets auto-approved straight into the dataset, or it gets routed to a human reviewer, who sees the flagged field highlighted right on the original text and can approve or fix it in seconds.

Swap the Domain Pack from "radiology" to "pathology," and the same five agents, same code, start producing tumor codes instead of X-ray findings. Nothing about the pipeline itself changes.

How we built it

We built the pipeline in Python using Google's Agent Development Kit (ADK) to structure the multi-agent flow, Gemini via Vertex AI as the reasoning engine behind every agent, and LangExtract for the actual source-grounded extraction step. The whole thing is wrapped in a single FastAPI service that handles both the processing API and a lightweight human-review UI, deployed on Cloud Run, backed by Cloud Storage for documents and datasets and Firestore for tracking the review queue.

We deliberately built it in phases: first we got the core Parser → Structuring → Verifier loop working correctly on synthetic radiology data, then added image alignment and the human review loop, then proved the "universal" claim by building a second Domain Pack for pathology coding and confirming the exact same pipeline code handled it with zero changes, and only then moved everything onto Google Cloud.

Challenges we ran into

Honestly, more of this project than we expected turned into a real production-engineering exercise rather than just AI pipeline work.

Early on, we caught a real architectural bug: our Structuring Agent's field-copying logic looked generic, but it was actually silently hardcoded to radiology's specific field names. If we'd shipped it as-is, our second Domain Pack would have quietly produced incomplete records while looking fine on the surface. We caught it, refactored the agent to be genuinely schema-driven, and used it as a lesson to grep the rest of the codebase for the same pattern.

Deploying to Google Cloud turned into its own saga. We hit a chain of IAM permission gaps one at a time — our service account was missing storage access, then Artifact Registry push access, then log-writing access, then finally Vertex AI access itself, each one only surfacing after we'd fixed the last one. We also discovered, the hard way, that /healthz is a reserved path on Google's infrastructure that silently 404s before ever reaching your container — no amount of rebuilding fixed it until we renamed the route entirely. And a rebuild that succeeded perfectly in the morning failed that same night on an unrelated machine, because our dependencies weren't pinned and a transitive package had shifted underneath us — we fixed it by generating a full lockfile so builds are actually reproducible now.

We also learned, late in the process, that LLM output isn't perfectly deterministic — the same report run twice can land on either side of a confidence threshold. Rather than hide that, we documented it as a known limitation and designed our demo around frozen, captured results instead of live re-runs.

Accomplishments that we're proud of

Getting the domain-swap to actually work, with zero pipeline code changes, is the thing we're most proud of — it's easy to claim a system is "universal," and much harder to prove it live. We also really like that our Verifier isn't just a rubber stamp: in one of our real test runs, it caught a case where the extraction step had pulled out "cardiomegaly" — an abnormal heart finding — from a report that actually said the heart was completely normal. That's the exact kind of subtle, dangerous error that would quietly corrupt a training dataset if nothing was checking for it, and our system caught its own mistake before it could.

What we learned

We came in thinking about this as an AI/agent design problem, and we left having learned just as much about the unglamorous realities of shipping something to real cloud infrastructure — IAM permissions, service account scoping, reproducible builds, and how quickly small assumptions ("this path is available," "this build will always resolve the same way") can turn into hours of debugging. We also learned to be honest about non-determinism instead of pretending our system is perfectly repeatable, which we think actually makes it more trustworthy, not less.

What's next for DataForge

We'd like to add a third, structurally different Domain Pack — something entirely outside medicine, like legal contract clause extraction — to push the "universal" claim even further. We also want to explore reducing the confidence-score variance we found near the auto-approve threshold, possibly through multi-pass consensus scoring for borderline cases, and eventually test the pipeline against real de-identified datasets like MIMIC-CXR instead of our synthetic samples.

Built With

• docker
• fastapi
• gemini
• google-adk
• google-artifact-registry
• google-cloud
• google-cloud-build
• google-cloud-firestore
• google-cloud-run
• langextract
• python
• uvicorn
• vertex-ai
• yaml