Shipcheck

An autonomous preflight agent that turns submission rules into executable checks, verifies repo and deployment evidence, and catches blockers before judges do.

Comment

Shipcheck

Not cleared until proven.

Inspiration

Software projects do not only fail because the code is broken. They can also fail at the point of submission.

A required architecture diagram is missing. A README does not explain how to reproduce the project. A deployment URL is unreachable. A submission claims a framework or model that the repository does not actually evidence. One mandatory rule is buried inside a long competition page and gets overlooked minutes before the deadline.

The frustrating part is that many of these failures are discoverable before submission, but the evidence is scattered across rules pages, repositories, documentation, runtime state, and submission claims.

Shipcheck started from a simple question:

Can this submission prove that it satisfies the rules?

The goal was not to build another chatbot that reads a rulebook and gives generic advice.

The goal was to build a preflight inspection system that turns rules into requirements, requirements into evidence checks, and evidence into a bounded release decision.

What it does

Shipcheck is an evidence-first autonomous preflight inspector for software submissions.

A user provides:

  1. a public competition or submission-rules URL;
  2. a public GitHub repository;
  3. an optional live deployment URL; and
  4. optional declared submission claims.

Shipcheck then interprets explicit requirements, inspects bounded repository and runtime evidence, checks declared claims, preserves requirements that still need human judgment, and returns one of three dispositions:

  • READY
  • NEEDS_REVIEW
  • HOLD

The result is presented as an Evidence Register, not a conversational answer.

Each inspected requirement can include:

  • evidence status;
  • severity;
  • observed repository or runtime evidence;
  • reasoning; and
  • a recommended action when remediation is needed.

A READY result is deliberately scoped. It means that no unresolved CRITICAL, HIGH, or MANUAL_REVIEW gate was found within the evidence Shipcheck could inspect.

It does not mean guaranteed eligibility, organizer acceptance, or competition success.

That distinction is central to Shipcheck.

How we built it

Shipcheck uses a hybrid agentic architecture.

The semantic part of the workflow is handled by a Google ADK rules agent using Gemini structured output.

The agent receives a bounded snapshot of the public rules page and converts explicit rule language into structured requirements.

The rest of the inspection is deliberately handled by deterministic application services:

  • a bounded GitHub inspector gathers repository metadata, tree entries, and selected source, configuration, and documentation files;
  • a static reproduction checker looks for safely observable setup and dependency evidence without executing arbitrary repository code;
  • an optional deployment verifier checks public reachability and bounded runtime evidence;
  • an evidence mapper connects structured requirements to concrete observations;
  • a claim checker separates unsupported claims from direct contradictions;
  • a deterministic disposition engine produces READY, NEEDS_REVIEW, or HOLD; and
  • optional Google Cloud Firestore persistence stores structured inspection audit records.

Gemini interprets natural-language rules.

It does not get unilateral control over the final compliance verdict.

Core Inspection Flow

Shipcheck end-to-end inspection flow

The end-to-end pipeline begins with the submission manifest and separates the inspection into multiple evidence channels.

Rules interpretation, repository inspection, runtime evidence, reproduction checks, and declared claims are evaluated independently before converging into a shared evidence and disposition layer.

This prevents one model response from becoming the entire inspection.

System Flowchart

Shipcheck operational flowchart

The operational flowchart shows one complete inspection from input validation through rule extraction, Gemini fallback, source grounding, repository and deployment inspection, evidence classification, final disposition, report generation, and optional Firestore audit persistence.

Architecture

Shipcheck runtime architecture

Shipcheck keeps the Google ADK agent focused on natural-language interpretation and surrounds it with bounded deterministic services.

This architecture gives the agent a real semantic responsibility without pretending that every repository check, HTTP request, contradiction decision, or severity calculation needs to be another LLM call.

Grounding Gemini instead of trusting it

Structured output makes model responses easier to consume, but valid JSON is not the same thing as valid evidence.

For uncached rules extraction, Shipcheck validates every extracted source_quote against the actual fetched rules text.

Formatting-only differences such as smart quotes, Unicode dashes, HTML entities, and whitespace normalization are tolerated.

Invented or paraphrased evidence is rejected.

Shipcheck also supports a bounded Gemini fallback chain. If one configured model is unavailable, quota-limited, times out, or cannot produce safely grounded output, the system can try another configured model instead of silently degrading the result.

The inspection report records which model actually completed the rules extraction and whether fallback was used.

Evidence is not the same as a claim

One of the most important design decisions in Shipcheck was separating missing evidence from contradiction.

Suppose a submission claims:

"This project uses Google ADK."

If Shipcheck cannot find sufficient evidence, that claim can be UNVERIFIED.

That does not automatically mean the claim is false.

A direct contradiction requires direct conflicting evidence.

Shipcheck therefore distinguishes between statuses such as:

  • VERIFIED
  • UNVERIFIED
  • MISSING
  • CONTRADICTED
  • MANUAL_REVIEW
  • NOT_APPLICABLE

The same principle applies to runtime evidence.

A Dockerfile can show deployment intent.

A Cloud Run command in documentation can show intended infrastructure.

Neither proves that a live Cloud Run service actually exists.

Configuration is evidence of configuration.

Runtime proof requires runtime evidence.

Human judgment is a valid output

Some submission requirements cannot safely be automated.

Age eligibility, geographic restrictions, originality, submission timing, video content, organization status, and subjective judging criteria can depend on information outside the repository.

Shipcheck does not force those requirements into artificial passes.

They remain:

MANUAL_REVIEW

That means uncertainty is part of the output rather than something hidden from the user.

Google Cloud integration

Shipcheck uses Google Cloud Firestore for optional inspection audit persistence.

A completed inspection can be persisted under its inspection ID with structured information such as:

  • final disposition;
  • repository;
  • rules source;
  • Gemini model used;
  • findings; and
  • inspection metadata.

The cloud evidence boundary is deliberately strict.

A Firestore write performed by Shipcheck normally proves something about Shipcheck's inspector runtime, not about an unrelated repository being inspected.

During configured Shipcheck self-inspection, Firestore evidence may count as target-project evidence only when the inspected repository matches the configured Shipcheck repository.

This prevents the inspector's own infrastructure from contaminating evidence about another project.

Self-inspection as proof of work

One of the final tests was intentionally recursive:

Shipcheck inspected Shipcheck.

The canonical self-inspection used the actual All Things Agentic Hackathon rules and the public Shipcheck repository.

The result was:

Result Count
Passed 7
Manual review 13
High 0
Warnings 13
Critical 0

Final disposition:

NEEDS_REVIEW

That result was useful precisely because it was not artificially green.

Shipcheck verified observable requirements such as the required Gemini generation, Google ADK implementation, public repository, README setup instructions, architecture evidence, and scoped Google Cloud Firestore usage.

Requirements involving entrant eligibility, submission timing, video properties, originality, and other human-controlled evidence remained under manual review.

The system did what it was built to do:

refuse to invent certainty where the evidence could not support it.

Challenges we ran into

Natural-language rules are not clean specifications

Competition rules mix technical requirements, eligibility conditions, legal language, examples, recommendations, judging criteria, and subjective requirements on the same page.

Trying to treat every sentence as automatically verifiable produced misleading results.

Shipcheck therefore separates checkable requirements from requirements that require manual review or are informational.

Structured model output was still not trustworthy enough

Gemini could produce syntactically correct structured output while normalizing, paraphrasing, or incorrectly reconstructing source text.

That led us to implement explicit source-quote grounding against the fetched rule snapshot.

The model interprets.

The source remains the authority.

Repository evidence was easy to overclaim

An early implementation could easily have become too optimistic.

A file named architecture.png is not automatically a valid architecture diagram.

A Dockerfile is not proof that Cloud Run is live.

A README sentence is not proof that the implementation behind the sentence exists.

The repository inspection rules became intentionally more conservative as the project matured.

Missing evidence is not contradiction

Early claim semantics risked treating:

"I could not find proof"

as equivalent to:

"I found proof that this is false."

Those are different statements.

The final evidence contract keeps UNVERIFIED, MISSING, and CONTRADICTED separate.

The inspector itself could contaminate evidence

Firestore introduced a subtle evidence problem.

Shipcheck genuinely uses Google Cloud, but that should not allow every repository inspected by Shipcheck to suddenly satisfy a Google Cloud requirement.

We therefore introduced an explicit boundary between inspector-runtime evidence and target-project evidence.

Arbitrary repository execution is unsafe

The obvious next step for reproduction checking would be to clone a repository and execute its setup commands.

For an untrusted public repository, that is not something a preflight tool should casually do.

Shipcheck currently performs bounded static reproduction checks and leaves arbitrary execution under manual review until a deliberately sandboxed runtime exists.

Real model providers fail

Agentic systems also have to deal with infrastructure reality.

Model endpoints can become quota-limited, unavailable, or slow.

Shipcheck therefore uses bounded model timeouts and explicit fallback behavior rather than assuming that one provider call will always succeed.

A failed model chain produces a failed inspection instead of fabricated evidence.

Accomplishments that we're proud of

The part we are most proud of is that Shipcheck does not pretend its LLM is an oracle.

The final prototype combines:

  • Google ADK + Gemini rules interpretation;
  • grounded source evidence;
  • bounded public GitHub inspection;
  • Python, JavaScript/TypeScript, Go, Java, and Kotlin source/configuration sampling;
  • static reproduction checks;
  • optional deployment verification;
  • explicit claim and contradiction semantics;
  • deterministic evidence severity and disposition logic;
  • an evidence-first non-chat interface;
  • Markdown report export;
  • optional Google Cloud Firestore audit persistence;
  • deterministic compliant and broken repository fixtures; and
  • unit, integration, and acceptance testing.

The final automated verification suite contains 62 passing tests, with Ruff static checks also passing.

The project also preserves a formal v01 verification set containing the canonical self-inspection report, repository smoke evidence, Firestore smoke evidence, pytest output, and Ruff output.

Shipcheck therefore does not only inspect other submissions.

Its own submission can leave an evidence trail too.

What we learned

The biggest lesson was that rule interpretation and compliance verification are different problems.

An LLM can be useful for understanding natural language.

Verification needs observable evidence.

A repository artifact can support a claim without proving runtime behavior.

A missing artifact can represent insufficient evidence without proving dishonesty.

A human-only requirement can remain unresolved without representing system failure.

We also learned that agentic architecture does not require turning every function into an agent.

Giving one agent a narrow semantic responsibility and surrounding it with deterministic services made Shipcheck easier to test, easier to reason about, and harder to fool with its own assumptions.

Another lesson was that uncertainty should not be treated as an embarrassing edge case.

Sometimes NEEDS_REVIEW is the most correct answer a system can give.

What's next

Shipcheck is still a hackathon prototype.

The next steps would be to add a deliberately sandboxed reproduction runner, stronger support for JavaScript-heavy rules pages and attached documents, authenticated private-repository inspection, richer deployment provenance, CI integration, and reusable competition-specific evidence adapters.

A future version could also act as a pre-submission gate for:

  • hackathons;
  • grants;
  • coursework;
  • open-source releases;
  • internal engineering reviews; or
  • any workflow where requirements must be proven before something is shipped.

The long-term question remains the same:

Does the project merely look ready, or can it actually prove that it satisfies the submission contract?

Shipcheck
Autonomous preflight for software submissions.
Not cleared until proven.

Built With

Share this project:

Updates

Submission history