NIST CSF Assessor

Inspiration

Every regulated company has to answer one question: how mature is our security programme, really?

The standard way to answer it is NIST CSF 2.0, the most widely adopted cybersecurity framework in the world. Six functions, twenty two categories, one hundred and six subcategories. The standard way to get that answer is to hire a consulting firm: fifty to a hundred thousand dollars, several weeks, and you do it again next year.

What makes it expensive is not paperwork. It is a hundred and six separate judgement calls. For each one somebody has to find the evidence, usually buried in policy documents nobody wrote with this framework in mind, decide whether a practice is merely defined or actually measured and managed, and then defend that score when a regulator asks why.

Because it costs that much, most organisations do it once a year. Which means that for the other eleven months they are making decisions from a snapshot that started going stale the day it arrived.

I wanted to change the cadence, not just the cost.

What it does

You upload your organisation's real policy documents. They get parsed, chunked, embedded, and indexed.

Then the interview, which is the part I am most pleased with. Instead of a fixed questionnaire, the system first measures how well your documents already cover each subcategory, then asks only about the gaps. It will not ask you something it can already read.

Six specialist agents, one per CSF function, then score all twenty two categories in parallel against a CMMI 1 to 5 rubric:

• Every score cites the specific document it came from.
• A second, independent model reviews each score for defensibility, because "the tool said so" does not survive a regulator's follow up question.
• Anything scored below sixty percent confidence is flagged as insufficient evidence rather than presented as solid.

And it remembers. Run it again in three months and the report tells you Govern moved from Tier 2 to Tier 3, and whether the gap it flagged last time actually got closed. That is the difference between a report and a programme.

The output is a board ready executive narrative plus PDF, HTML, and Excel exports.

How we built it

The orchestrator runs on Vertex AI Agent Runtime under a native SPIFFE agent identity, not a shared service account. It dispatches over Pub/Sub to six parallel Cloud Run Job executions. Firestore is the source of truth throughout, which is why a dropped connection never loses a run.

That split is deliberate. A six way fan out of short batch jobs belongs on Cloud Run Jobs; Agent Runtime is built for a long lived orchestrator. Putting the specialists on Agent Runtime would have been the wrong tool.

Three workloads, three least privilege identities. The public UI holds five permissions and cannot read uploaded evidence in Cloud Storage, cannot publish to Pub/Sub, and cannot trigger a job. The component anyone on the internet can reach is the one that can do the least.

Google's Model Armor screens untrusted input for prompt injection and jailbreak at all three tiers, and a deterministic scrubber redacts emails, national IDs, card numbers, and IP addresses before any text reaches a model.

Built with Google ADK, Gemini 3.5 Flash for ingest and scoring, Gemini 3.7 Flash for the judge and narrative, text-embedding-004, FastAPI, Streamlit, Qdrant Cloud, and OpenTelemetry into Cloud Trace.

Challenges we ran into

Firestore rejects certificate bound Agent Identity tokens. With Google's default policy on, every call failed with a 401. Rather than guess, I built a throwaway probe that tested four transports independently. Firestore failed on sync gRPC, async gRPC, and plain HTTPS REST, while Pub/Sub succeeded on the same credential in the same container. That proved it was a service side gap rather than a transport problem on my end, and it meant the library refactor I had been about to spend hours on would have changed nothing.

Agent Gateway has a latency ceiling I could not design around. Gateway mediated runs capped at roughly 305 seconds, six times. The control run that made it a measurement rather than a guess was the ungoverned engine running the same six function assessment for 590 seconds and completing fully.

Google's own documentation was wrong in three places, each costing real time:

• registries is documented as Optional but is mandatory for egress, and omitting it denies 100% of traffic with no error naming the cause.
• The documented resource format returns HTTP 400.
• There is no ENFORCED value at all: enforcement is the absence of a field.

The bugs that mattered came from running the system, not reading it. A sys.path bootstrap that ran after the imports it was meant to enable, crashing the deployed UI. A blocking call inside an async function that pinned the event loop, so the gateway served nothing at all while an assessment was in flight. And a Pub/Sub consumer that acknowledged every message it pulled, silently destroying concurrent runs' completion messages and stranding them until a 900 second timeout.

Accomplishments that we're proud of

I measured accuracy instead of asserting it. A cybersecurity practitioner independently scored all one hundred and six subcategories without seeing my output. Compared blind:

Metric	Result
Tier agreement across all six functions	100%
Subcategories within one CMMI level	100% (106 / 106)
Mean absolute error	0.39

The golden set ships in the repository so anyone can re run it.

Five GEAP components run natively in the live path, not as labels on hand rolled equivalents: Agent Runtime, Agent Identity, Agent Registry, Model Armor, and Observability. The Coordinator was auto registered by Agent Runtime itself, which is stronger evidence of real adoption than anything I could have registered by hand.

The Judge caught a real bug in my own pipeline. Not a staged demo: it flagged a genuine internal inconsistency where a numeric score did not match the written reasoning next to it, on real output.

Every gap has a documented reason. Agent Gateway is built and measured with 50 of 50 requests matching a registered destination, then deliberately detached with the numbers to justify it. Memory Bank is hand rolled because native agentengine:// is conversational memory, a different data model. I would rather ship a defensible "no" than a quiet omission.

What we learned

Native adoption means finding out where the platform's limits actually are, and the honest version of that is often "I measured this and here is the constraint" rather than a clean checkbox.

I also learned to distinguish carefully between what I proved and what I inferred. I can prove the Agent Gateway ceiling exists, because I have six capped runs against a 590 second control. I cannot prove why, and I say so. That distinction turned out to matter more than any individual feature.

And a specific technical lesson worth carrying: HTTP 200 is not a health check. The deployed Streamlit UI returned 200 for hours while the app script was crashing behind it, because the server was answering and the script was not.

What's next for NIST CSF Assessor

• Multi company benchmarking, so accuracy becomes a benchmark rather than a validation against one expert.
• Per destination enforcement testing on the Agent Gateway path, which needs the latency ceiling resolved first.
• Calibrating the systematic plus one I see on Identify subcategories. I deliberately did not tune that away, because with tier agreement already at six out of six it would mean overfitting to a single practitioner's judgement rather than improving accuracy.
• Moving network authentication from application level API keys to IAP, so enforcement sits at the network layer rather than in the application.

Built With

• agent-identity
• agent-registry
• agent-runtime
• cloud-run
• cloud-storage
• cloud-trace
• docker
• fastapi
• firestore
• gemini
• gemini-3.5-flash
• gemini-3.7-flash
• google-adk
• google-cloud
• model-armor
• multi-agent
• opentelemetry
• pub-sub
• pydantic
• python
• qdrant
• rag
• streamlit
• vertex-ai