PCN Triage Orchestrator

ADK+Gemini 3.5 multi-agent pipeline: Gmail/Pub-Sub→Eventarc→3-stage Triage/Resolution/Action on Cloud Run. Reads scanned PCNs natively, queries Firestore, ships GitHub PRs. Idempotent, keyless.

Comment

Inspiration

Hardware and firmware teams deal with a constant, unglamorous chore: Product Change Notifications. A manufacturer discontinues a part, and someone has to read the PDF, figure out if it's even used in the current design, find a pin-compatible replacement, update the firmware header, and file the paperwork. It's exactly the kind of multi-step, low-visibility work that eats real engineering time and rarely gets automated — because doing it wrong (fabricating a "compatible" replacement, say) is worse than not doing it at all. That tension — automate it fully, but never let it guess — is what we set out to build.

What it does

The PCN Triage Orchestrator watches a Gmail inbox for incoming PCN emails, reads any attached PDF natively through Gemini's multimodal capability — including genuinely scanned, image-only documents with zero extractable text — cross-references every affected part number against a Firestore inventory, and autonomously opens GitHub Pull Requests updating firmware HAL headers, plus generates Engineering Change Order PDFs. No human touches anything between an email arriving and a PR appearing on GitHub. If a part isn't in inventory, it says so explicitly instead of inventing a fix.

How we built it

System Architecture Every internal GCP component and external service, and how data flows between them.

Two decoupled Cloud Run services. pcn-ingestor watches Gmail via users.watch() + Pub/Sub push, checks a sender allowlist, and drops verified PDFs into a GCS bucket.

Ingestion Sequence Gmail push notification → OIDC verification → sender allowlist check → Gmail fetch → GCS upload.

That upload fires an Eventarc trigger into pcn-agent, which runs a three-stage pipeline built on Google ADK 2.0 and Gemini 3.5 Flash: a Triage agent reads the PDF natively and extracts part numbers, a Resolution agent checks each against Firestore, and an Action agent opens GitHub PRs and generates ECO PDFs for anything it found.

Multi-Agent Pipeline Flow Every terminal outcome the pipeline can reach — including the two idempotency and cost-guard exit paths that run before Stage 1 ever fires.

Every run's outcome lands as a structured Firestore document — not a prose blob — so the whole system is queryable, not just observable in logs.

Both services run fully locked down (no public ingress, OIDC-only), fully keyless (Application Default Credentials everywhere, no service account key files), and idempotent against Pub/Sub and Eventarc's at-least-once delivery guarantees.

Challenges we ran into

Every one of these was caught in live production testing, not a code review:

  • The agent hallucinated a part number. Early on, it only had a bare GCS URI as text — no way to actually read the PDF — and invented a plausible-sounding part number instead of failing. Nine PRs went out against fabricated data before we caught it and switched to true multimodal PDF input.
  • ADK's SequentialAgent couldn't resume mid-pipeline. We assumed an outer retry loop would retry only a failed stage. It restarted everything from scratch instead — including the expensive multimodal read. We rebuilt orchestration as three independent agents sharing one session, with explicit state handoff in code.
  • Pub/Sub redelivered the same event and doubled our spend. At-least-once delivery is documented Google behavior, not a bug — but it meant one GCS upload triggered the full pipeline twice. Fixed with a Firestore idempotency check before Stage 1 even runs.
  • The LLM's own function-calling schema mangled a filename. Passing {"hal_bme280.h": content} as a dict let the schema layer strip the dot from the key, producing hal_bme280_h — and once, a zero-width-space-corrupted filename. The fix was structural: make the filename a value, never a dict key.

Accomplishments that we're proud of

A real, working autonomous pipeline — not a demo shaped around a happy path. It correctly handles a genuinely scanned zero-text-layer PDF, a multi-page document naming two different parts (one resolved, one honestly reported as unresolved, in the same run), and a duplicate event delivery caught and skipped before it wastes a single LLM call. Every claim in our README is backed by a real screenshot of the actual running system, not an illustration.

What we learned

That "the pipeline completed successfully" and "the pipeline produced correct output" are different claims — several of our bugs were only caught because we opened the actual GitHub PR diff, not just the exit logs. And that documenting why an architectural decision was made, including the wrong turn that preceded it, is worth more than documenting only the final state.

What's next for PCN Triage Orchestrator

Automating Gmail watch renewal via Cloud Scheduler (currently a manual weekly step), moving to the GitHub Trees API for atomic multi-file commits when a PCN touches more than one HAL file, load-testing under concurrent PCN volume, and extending part-impact analysis beyond a single HAL header to a full BOM cross-reference.

Built With

  • fastapi
  • firestore
  • gemini
  • gmail-api
  • google-adk
  • google-cloud
  • google-cloud-eventarc
  • google-cloud-pubsub
  • google-cloud-run
  • mermaid
  • oauth2
  • pygithub
  • python
  • reportlab
  • vertex-ai
Share this project:

Updates

Submission history