• 

  CMMC Scout

CMMC Scout

Inspiration

Defense contractors are facing a compliance cliff. Starting in 2025, you can't bid on DoD contracts without CMMC Level 2 certification—but most small manufacturers have no idea if they'll pass. A formal C3PAO assessment costs $15-50K, and if you fail, you're out that money plus months of remediation time before you can try again.

I've spent years as a CMMC consultant watching companies gamble on assessments they weren't ready for. The 110 controls in NIST SP 800-171 are dense, technical, and interconnected. Business owners don't know what "limit unsuccessful logon attempts" actually requires in practice, or that their email-based access approvals won't cut it.

I wanted to build an AI agent that could conduct the pre-assessment conversation I have with clients—asking the right follow-up questions, explaining what compliance actually looks like, and generating a clear remediation roadmap. Not to replace human consultants, but to make that first "where do we stand?" conversation accessible to contractors who can't afford us yet.

What it does

CMMC Scout is an AI agent that conducts interactive compliance assessments for CMMC Level 2 certification.

The user experience:

1. Log in via Auth0 with role-based access (assessor vs. client)
2. Select a security domain to assess (e.g., Access Control)
3. The agent asks contextual questions about each control
4. User responds with their current practices
5. Agent scores each control (Met / Partially Met / Not Met) with plain-English explanations
6. Generate a gap report with prioritized remediation steps, estimated costs, and timelines

What makes it production-ready:

• Every assessment decision streams to Redpanda for audit logging—because compliance work needs receipts
• Akka actors manage assessment state, so sessions can recover from failures
• Comet tracks prompt performance across controls for continuous improvement
• Auth0 ensures sensitive assessment data stays protected

How I built it

Architecture decisions:

I chose an actor-based model with Akka because assessments are inherently stateful, long-running conversations. Each assessment session is an actor that maintains state across multiple control evaluations. If the system crashes mid-assessment, the actor can recover. This maps naturally to how real assessments work—you don't lose your progress.

Redpanda handles event streaming for every significant action: assessment started, control evaluated, gap identified, report generated. This gives me a complete audit trail (essential for compliance work) and enables real-time dashboards for consultants managing multiple client assessments.

I used Auth0 for authentication because security-by-design matters when you're handling sensitive compliance data. Role-based access control means clients only see their own assessments while consultants can manage multiple clients.

Comet tracks my prompt engineering experiments. Different controls need different assessment approaches—some are yes/no technical checks, others require nuanced conversation about policies and procedures. Comet lets me version prompts and measure which approaches yield the most accurate assessments.

Tech stack:

• Python / FastAPI for the backend
• Claude API for the assessment agent
• Akka for workflow orchestration
• Redpanda for event streaming
• Auth0 for authentication
• Comet for ML observability
• PostgreSQL for persistence

Data: I loaded the 22 Access Control domain controls from NIST SP 800-171 Rev 2, including assessment objectives and discussion points. The agent uses this as context when formulating questions and scoring responses.

Challenges I ran into

Akka learning curve: Actor-based programming is a different mental model. I meant to spend time early with Hugh McKee at the Akka booth getting the right patterns for my use case. The key insight was treating each domain assessment as a child actor of the session actor, which simplified state management considerably.

Scoring consistency: Getting the agent to score controls consistently was harder than expected. "Partial" compliance is subjective—one user's "we have a policy" might be a fully documented procedure, another's might be an informal understanding. I iterated on few-shot examples in my prompts and tracked results in Comet until scoring stabilized.

Balancing depth vs. time: A real CMMC assessment takes days. I had to design the agent to ask enough follow-up questions to make accurate assessments without turning a demo into a 30-minute interrogation. I settled on 2-3 contextual follow-ups per control.

Event schema design: Deciding what to stream to Redpanda and at what granularity took iteration. Too granular and I'm drowning in events; too sparse and I lose the audit trail.

Accomplishments that I'm proud of

• All four vendor integrations working together: Auth0, Akka, Redpanda, and Comet aren't just checked boxes—they each solve a real problem in making this production-ready.

• Explainable assessments: The agent doesn't just score controls, it explains why something is partial or non-compliant in plain English that a business owner can understand.

• Audit trail from day one: Every assessment decision is logged and replayable. This is table stakes for compliance tooling but often added as an afterthought.

• Domain expertise encoded in prompts: The agent asks the follow-up questions a human CMMC consultant would ask, not generic security questions.

What I learned

• Actor models are powerful for workflow orchestration: Once I got past the learning curve, Akka made state management much cleaner than managing it manually. I'll use this pattern again.

• Event streaming belongs in more projects: Redpanda wasn't just for audit logging—being able to replay events made debugging the agent's decision-making much easier.

• Prompt engineering needs observability: I couldn't have achieved consistent scoring without Comet tracking my iterations. Prompt engineering without metrics is just vibes. I will definitely use Comet again and again.

What's next for CMMC Scout

Immediate:

• Expand beyond Access Control to all 14 CMMC domains (110 controls)
• Add evidence upload capability (policy documents, screenshots)
• POA&M (Plan of Action & Milestones) generation

Short-term:

• Package as SaaS for CMMC consultants to white-label
• Multi-user assessments (different people answer for their domains)
• Progress tracking over time as gaps are remediated

Long-term:

• Integrate with Microsoft Graph API to automatically pull evidence from Azure/M365 (Conditional Access policies, Intune configs, etc.)
• Pre-built integrations with GRC platforms
• Formal validation against C3PAO assessment outcomes

The DIB sector has 300,000+ contractors who need CMMC certification. Most can't afford consultants. This agent can democratize access to compliance readiness.

Built With

• akka
• auth0
• claude
• comet
• fastapi
• postgresql
• python
• redpanda

Built With

• akka
• auth0
• claude-api
• comet
• docker
• fastapi
• postgresql
• python
• redpanda