Inspiration

Large infrastructure teams make changes that can affect hundreds of services. Traditional infrastructure tests can verify that a cluster, network, or storage system is configured correctly, but they cannot always predict whether an application depends on an obscure behavior being changed.

These dependencies are often scattered across documentation, deployment manifests, source code, and people’s memories. As organizations grow, coordinating every infrastructure change through meetings and manual reviews becomes slow, risky, and difficult to scale.

We built Snowy to turn this fragmented knowledge into automated, evidence-backed compatibility checks.

What it does

Snowy evaluates infrastructure changes against every registered service before they reach production.

It handles two important scenarios:

Known tribal knowledge: Service teams can declare requirements they already know about, such as fixed IP addresses, storage performance, networking behavior, or other operational assumptions.
Hidden dependencies: Snowy uses AI to compare the proposed infrastructure change with each service’s actual code, Kubernetes manifests, and runtime configuration. This helps discover dependencies that were never documented.

Each service evaluates the change inside its own repository and returns a Compatibility Attestation. Snowy aggregates these results into one enforceable report on the infrastructure Dashboard with results for each change or as an comment on the PR.

This also bridges the knowledge gap between teams. Application developers should not need to understand every platform mechanism, such as how AppArmor, seccomp, or cluster policies work. Infrastructure teams should not need to understand every application’s startup behavior. Snowy connects the platform-level change to its practical application-level consequences.

Snowy: End-to-End Infrastructure Change Workflow

Snowy allows an infrastructure team to test a proposed platform change against every registered application repository before merging it.

The infrastructure repository does not attempt to understand every application centrally. It distributes the proposed change to each service, where the service's own code, manifests, configuration, explicit requirements, and hidden runtime dependencies can be evaluated.

Live Project Links

Component	Link
Contract dashboard	Snowy Contract Radar
Contract registry	demo-contract-registry
Infrastructure repository	demo-infra-platform
Search service	demo-service-search
Node Inspector service	demo-service-node-inspector
Live demonstration PR	Infrastructure PR #3
Exact infrastructure diff	PR #3 changed files
Final infrastructure workflow	Actions run, attempt 3
Search evaluation	Search workflow run
Node Inspector evaluation	Node Inspector workflow run
Final PR report	Snowy's aggregated PR comment

Workflow Overview

flowchart LR
    A["Infrastructure PR #3"] --> B["Trusted infra workflow"]
    B --> C["Read registry routing index"]
    C --> D["Dispatch exact change to Search"]
    C --> E["Dispatch exact change to Node Inspector"]

    D --> F["Search evaluates its code, manifests and requirements"]
    E --> G["Node Inspector evaluates its code, manifests and requirements"]

    F --> H["Search artifact: PASS"]
    G --> I["Node artifact: FAIL - NODE-SEC-001"]

    H --> J["Infra fan-in aggregator"]
    I --> J

    J --> K["PR compatibility report"]
    K --> L["Merge blocked"]

1. The Infrastructure Team Opens a PR

The demonstration starts with Infrastructure PR #3.

The PR changes two cluster-wide security defaults:

spec:
  seccomp_default: true
  apparmor_default: true

The complete diff contains only two changed values.

This is exactly the kind of change that appears small to an infrastructure team but can affect application behavior in ways that are not visible from the infrastructure repository alone.

2. The Infrastructure Workflow Starts

The PR triggers the infrastructure repository's Check service contracts workflow.

For security, it uses pull_request_target and executes only trusted workflow code from the base branch. The candidate PR code is never executed with privileged credentials. Its description and diff are treated only as input data.

The workflow creates a unique correlation ID containing:

Infrastructure repository
Pull request number
Candidate commit SHA
GitHub Actions run ID
Workflow attempt

For the final live run, the correlation ID is:

samrathpalsingh-demo-infra-platform-pr3-3b6cf8b4ee00-run29876530132-3

This prevents results from older runs, other pull requests, or unrelated repositories from being accepted accidentally.

3. Snowy Discovers the Registered Services

The infrastructure workflow reads the registry's routing index.

The registry currently contains two enabled service consumers:

The registry identifies where each service lives and which workflow should evaluate an incoming infrastructure change.

The live registry can also be viewed through the Snowy Contract Radar dashboard.

The registry is a routing and contract-publication layer. It does not centrally evaluate application code.

4. The Change Is Sent to Every Service Repository

The infrastructure workflow sends a correlated repository_dispatch event to each registered service.

The event includes:

Infrastructure repository
PR number
Candidate commit SHA
PR title and description
Correlation ID
Registry repository and revision
Service identity

Credentials and API keys are never included in the payload.

Each service workflow runs independently, inside the service's own GitHub repository:

This is what makes the design scalable: adding another service does not make the infrastructure repository responsible for understanding that service.

5. Each Service Evaluates the Change Locally

Each service checks out its own trusted default branch and gathers:

Application source code
Kubernetes manifests
Configuration
Documentation
Owner-declared requirements
Its published service contract
The exact infrastructure PR diff and description

The evaluation therefore covers two important categories.

Scenario A: Explicit or Tribal Knowledge

Service owners can document requirements that are known to the team but are difficult for the infrastructure team to discover independently.

The Search service contains requirements such as:

Preserve allow-listed NAT egress IP addresses
Preserve a legacy public ingress IP
Maintain at least 3,000 IOPS for its storage

These requirements are visible in the Search infrastructure requirements.

This converts tribal knowledge into versioned, executable evidence.

Scenario B: Hidden Code and Deployment Dependencies

Snowy also examines application code and deployment configuration for dependencies that were never explicitly communicated to the infrastructure team.

Node Inspector contains a startup operation that invokes:

mount -t tmpfs tmpfs /mnt/runtime-check

Evidence can be inspected directly:

Its Deployment requests CAP_SYS_ADMIN and does not select a compatible AppArmor or seccomp profile.

A service team may reasonably focus on application behavior rather than cluster-wide security mechanisms. Snowy connects the infrastructure team's platform knowledge with concrete evidence from the application repository.

6. Search Returns PASS

The Search service evaluated the change in its own repository:

View the Search workflow run

Result:

PASS
Risk: none

The proposed AppArmor and seccomp defaults do not violate Search's networking, ingress, storage, code, or deployment requirements.

This is important: Snowy does not reject every infrastructure change. It produces service-specific evidence.

7. Node Inspector Returns FAIL

Node Inspector evaluated the same infrastructure change independently:

View the Node Inspector workflow run

Result:

FAIL
Risk: critical
Rule: NODE-SEC-001

Reason:

The service's startup command requires mount(2) and CAP_SYS_ADMIN. Common runtime-default AppArmor and seccomp confinement denies this operation, so the container can fail during startup.

Suggested remediation:

Provide and test a narrowly scoped service-specific profile
Grant a reviewed temporary exception
Remove the mount(2) dependency before enabling the cluster-wide default

The failing service workflow is intentional. It demonstrates that Snowy found a real incompatibility before deployment.

8. Each Service Returns a Correlated Artifact

Each service uploads a small machine-readable result.json artifact.

The artifact contains:

Correlation ID
Service ID
Repository
Owner
PASS or FAIL
Risk level
Summary
Violated rule IDs
Evidence
Reason
Recommended remediation

The infrastructure workflow accepts an artifact only when its correlation ID and service ID match the expected values.

This prevents a stale or unrelated result from being counted.

9. The Infrastructure Workflow Aggregates the Results

The original infrastructure workflow polls the service repositories for their correlated artifacts.

The final live run is available here:

View the final infrastructure Actions run

It received:

Service	Result	Risk
Search API	PASS	None
Node Inspector	FAIL	Critical

The infrastructure workflow then generated:

A machine-readable aggregate artifact
A human-readable report on the pull request
A failing merge-gate status

10. Snowy Posts One Report on the Infrastructure PR

The final report is posted where the platform engineer is already working:

View Snowy's aggregated report on PR #3

The report states:

1 of 2 service repositories rejected this infrastructure change.

It includes:

A link to each service repository
A link to each service workflow run
PASS or FAIL
Risk severity
Evidence
Violated rules
Remediation guidance

The judge does not need to search through multiple repositories manually. The infrastructure PR becomes the single decision point.

11. The Merge Gate Is Blocked

Because Node Inspector returned a critical incompatibility, the infrastructure workflow exits with a failure.

The red status is the expected successful outcome of this demonstration: Snowy prevented an unsafe infrastructure change from being merged.

If all registered services returned PASS, the same workflow would complete successfully and permit the change to continue through the normal review process.

Why the Dashboard and PR Show Different States

The Snowy dashboard represents the currently deployed contract baseline.

The dashboard shows both services as healthy because the current platform baseline still has AppArmor and seccomp defaults disabled.

PR #3 represents a proposed future state. Its compatibility failure appears on the pull request and blocks that proposal before it becomes the deployed baseline.

This distinction is intentional:

Dashboard: compatibility with the current platform
Pull request report: compatibility with a proposed platform change

What This Demonstrates

Snowy successfully demonstrates that a large infrastructure team can:

Submit one platform change
Discover all registered application consumers
Send the change to each owning repository
Test explicit tribal knowledge
Discover hidden dependencies from code and deployment configuration
Run evaluations independently under service ownership
Correlate results safely
Aggregate evidence into one infrastructure report
Block an incompatible change before production

The infrastructure team does not need to understand every service's implementation, and service teams do not need to become experts in every platform-level mechanism. Snowy connects those two domains using distributed, executable evidence.

How we built it

Snowy is implemented primarily in Go and organized into four independent repositories to show how it works:

An infrastructure repository that initiates and aggregates compatibility checks.
Two service repositories that evaluate changes using their own code and configuration.
A Git-backed registry containing service ownership, routing information, and published requirements.

When an infrastructure pull request is opened, Snowy reads the registry and dispatches a correlated request to every participating service repository.

Each service workflow then:

Checks out its trusted default branch.
Fetches the infrastructure pull-request description and diff.
Loads its owner-defined requirements.
Collects relevant code, manifests, and configuration.
Uses AI to produce a structured compatibility verdict.
Applies deterministic safeguards for critical known and unknown patterns.
Returns a machine-readable result to the infrastructure workflow.

The infrastructure workflow validates and aggregates every response. The pull request passes only when all participating services accept the change.

To demonstrate hidden-dependency detection, one service runs a command that requires a Linux mount operation. A proposed cluster-wide security-policy change looks valid from the infrastructure perspective but is rejected after Snowy connects it to the service’s actual startup behavior.

Challenges we ran into

The biggest challenge was designing a system that could scale without centralizing every service’s code and knowledge inside the infrastructure repository.

We also had to solve several workflow and security problems:

Safely coordinating workflows across multiple GitHub repositories.
Correlating every service result with the correct infrastructure pull request.
Handling missing, invalid, or timed-out responses without accidentally approving a change.
Giving AI enough context without creating excessively large prompts.
Treating pull-request descriptions, diffs, and repository files as untrusted input.
Combining semantic AI analysis with deterministic checks for critical dependencies.
Keeping service teams responsible for application behavior while allowing infrastructure teams to retain ownership of platform mechanisms.

We ultimately used repository dispatch events for fan-out and correlation-specific workflow artifacts for fan-in.

Accomplishments that we're proud of

We built a functioning product which works in a distributed rather than a centralized way.

Snowy currently demonstrates:

Independent evaluation inside each service repository.
Detection of both documented and previously hidden dependencies.
AI-powered analysis grounded in real code, manifests, and owner-defined requirements.
A fail-closed infrastructure merge gate.
Evidence-backed results with rule IDs, severity, reasoning, and remediation.
A registry-driven design where adding another service does not require changing the infrastructure workflow.
Local deterministic testing without consuming AI quota.
A realistic security-policy scenario that affects one service while leaving another unaffected.

We are particularly proud that Snowy does not simply label an infrastructure change as “risky.” It identifies which service is affected, what implementation detail creates the dependency, and what must happen before the change can proceed safely.

What we learned

Infrastructure compatibility is not just an infrastructure problem—it is where platform assumptions meet application reality. And sometimes that meeting is less “smooth integration” and more “why did production suddenly catch fire?” Snowy taught us that AI works best as a detective, not a fortune-teller. We give it real evidence: owner-defined requirements, trusted code, deployment manifests, structured outputs, deterministic safeguards, and fail-closed workflows. The biggest lesson? Compatibility must follow ownership. Infra teams coordinate the investigation, but each service repository knows where its own skeletons are buried. Snowy simply asks every service: “Will this infrastructure change break you—and can you prove it?”

The answers return as one clear platform decision, with fewer meetings, less tribal knowledge, and considerably less guessing.

What's next for Snowy

Future improvements

Today, Snowy evaluates infrastructure changes against service code and configuration. Next, we want compatibility checks to work in both directions: service changes should also be evaluated against the current infrastructure setup, policies, limits, and capabilities.

Snowy could also recursively fan out from each service to its dependencies, then fan the results back in:

An infrastructure or service change starts an evaluation.
Snowy checks the change against the opposite side of the boundary.
Each affected service evaluates its own code, manifests, and requirements.
Services forward relevant impacts to their dependencies.
Snowy aggregates every verdict into a complete dependency and blast-radius map.

Over time, this could create a living map of how services, APIs, databases, queues, and platform components interact—revealing direct failures, hidden dependencies, and cascading risks.

The goal is to evolve Snowy from a compatibility checker into a two-way change intelligence system that answers:

What might break, what depends on it, and which teams need to act?

Snowy