-
-
Verified security controls: GitHub OIDC, temporary AWS credentials, Secrets Manager, least-privilege IAM, MCP and HTTPS.
-
Operational Memory Graph linking the active incident to the top resolved memories retrieved from CockroachDB vector search.
Inspiration
Production incidents rarely happen only once.
Teams often solve the same class of failure multiple times, but the useful knowledge is buried in old tickets, work notes, troubleshooting history, and individual experience. When the next outage starts, engineers frequently begin the investigation almost from zero.
That means valuable time is lost rediscovering root causes and remediation steps that the organization has already validated before.
Agentic Incident Memory was built around a simple question:
What if every resolved incident could become trusted operational memory for the next one?
The goal was not to build an autonomous AI that is free to search, decide, and modify production systems.
The goal was to build a controlled operational-memory layer that helps engineers recover verified knowledge faster, while preserving provenance, lifecycle rules, privacy boundaries, and application-owned security controls.
The result is a system designed to reduce investigation time without lowering the security bar.
What it does
Agentic Incident Memory turns resolved incidents into persistent, searchable operational memory.
When a new active incident appears, the application follows a controlled investigation pipeline:
- The current incident first passes through a deterministic Privacy Guard.
- Configured direct identifiers are removed before they can reach embeddings, durable operational memory, or the main Amazon Bedrock Investigator.
- If redaction occurs, a separate Bedrock Privacy Guard reviewer receives only the already-sanitized text and returns a narrow
PASSorREVIEW_REQUIREDdecision. - A
REVIEW_REQUIREDresult stops the processing path before embedding generation or Investigator inference. - Amazon Titan Text Embeddings V2 converts the sanitized incident symptoms into a semantic embedding.
- CockroachDB performs vector similarity search against trusted resolved operational memories.
- The application retrieves up to five relevant historical incidents with verified root causes and resolutions.
- Retrieved evidence is validated and sanitized again before it can enter the Investigator context.
- Amazon Bedrock receives only the controlled current-incident context and application-selected historical evidence.
- The operator receives a grounded diagnosis, recommended actions, supporting incidents, execution telemetry, and provenance.
The result can be explored in the public Agentic Incident Command Center or used directly from a ServiceNow incident.
Active incidents are intentionally treated as untrusted investigation context.
They do not become operational memory simply because an AI generated a hypothesis.
Only resolved or closed incidents can enter the trusted memory lifecycle.
The current reviewed demo corpus contains:
- 60 synthetic ServiceNow incidents
- 50 resolved operational memories
- 10 active investigation scenarios
- 10 semantic failure clusters
- up to 5 supporting historical incidents per investigation
All demo incidents, identities, and operational data are synthetic and fictional.
How we built it
CockroachDB is the persistent operational-memory layer of the system.
The architecture deliberately keeps semantic retrieval, lifecycle rules, application control, AI generation, and enterprise workflow integration separated into clear trust boundaries.
CockroachDB Distributed Vector Indexing
Resolved incidents are transformed into semantic embeddings and stored together with their verified operational context, including root cause and resolution.
CockroachDB vector similarity search retrieves the resolved incidents most relevant to the current failure.
This keeps transactional operational data and semantic memory together rather than introducing a completely separate vector database.
The model itself does not decide which historical incidents become evidence.
Application code controls:
- memory scope
- retrieval limits
- service and environment filters
- vector query construction
- validation of returned records
- evidence selection
- supporting incident IDs
This creates deterministic retrieval around the probabilistic AI layer.
Privacy-aware operational-memory ingestion
Persistent agentic memory introduces an important security question:
What information are we willing to remember?
Protecting only the final LLM prompt is not sufficient if sensitive information has already been embedded and stored as long-term semantic memory.
For that reason, privacy controls are applied before memory creation.
When a resolved incident is synchronized into operational memory, the application sanitizes fields used for embedding and persistence, including:
- title
- symptoms
- root cause
- resolution
- service
- environment
- tags
- nested metadata values
Configured direct identifiers are therefore removed before embedding generation and before CockroachDB persistence.
Fields whose meaning controls memory identity or tenancy are treated differently. If sensitive identifiers are detected in control-plane fields such as memory scope or source identity, the request is rejected rather than silently changing security semantics.
This provides a privacy boundary before durable semantic memory exists instead of attempting to clean sensitive information after it has already entered the memory store.
Privacy Guard with separation of duties
The Privacy Guard is intentionally split into two responsibilities.
First, deterministic application code performs the actual redaction.
It currently provides bounded detection for configured direct identifiers such as email addresses and explicitly identified name and phone fields.
Only after deterministic redaction has already taken place can a secondary Bedrock Privacy Guard reviewer be invoked.
That reviewer receives sanitized text only.
It cannot:
- access CockroachDB
- select SQL
- change memory scope
- select supporting incidents
- write operational memory
- reconstruct removed values
- call the main Investigator
- modify ServiceNow
Its responsibility is deliberately narrow: return either PASS or REVIEW_REQUIRED.
If it returns REVIEW_REQUIRED, the request stops before Titan embedding generation and before the main Bedrock Investigator.
If the secondary review is temporarily unavailable, deterministic redaction remains enforced and the degraded audit state is reported without restoring removed values.
This creates a real separation of duties between deterministic privacy enforcement and AI-assisted privacy review.
The project intentionally describes this as a bounded direct-identifier control, not as universal PII, PHI, DLP, or regulatory-compliance detection.
CockroachDB Cloud Managed MCP
The backend accesses operational memory through a constrained CockroachDB Cloud Managed MCP integration.
MCP operations are allowlisted and selected by application code rather than freely chosen by the language model.
The model never controls:
- SQL
- database operations
- memory scope
- retrieval count
- vector query structure
- incident identity
- which supporting records are returned
This prevents the language model from becoming the database control plane.
AWS architecture
The agentic backend is deployed on AWS and uses:
- AWS Lambda for serverless incident investigation and memory ingestion
- Amazon API Gateway for controlled application APIs
- Amazon Bedrock for evidence-grounded incident diagnosis and recommendations
- Amazon Titan Text Embeddings V2 for semantic incident embeddings
- Amazon Bedrock for the separate Privacy Guard reviewer
- AWS Secrets Manager for integration credentials
- AWS IAM / STS for least-privilege temporary credentials
- Amazon CloudWatch for AWS-side operational visibility
- AWS SAM for infrastructure deployment
The application architecture separates AI responsibilities rather than using one unrestricted model for every task.
The Privacy Guard reviewer has a narrow validation role.
The Investigator receives only sanitized current-incident context and application-selected historical evidence.
Neither model controls persistence, SQL, scope, or evidence selection.
GitHub OIDC and short-lived AWS access
GitHub Actions uses:
GitHub OIDC → AWS STS → temporary AWS credentials
The automation flow therefore does not require long-lived AWS access keys stored in the repository.
The OIDC role is intentionally constrained to the resources required by the reviewed automation path.
Integration credentials remain outside source code in AWS Secrets Manager.
Enterprise workflow
A ServiceNow scoped application allows an operator to run Analyze with Agentic Memory directly from an active incident.
The enterprise flow is:
ServiceNow Incident → API Gateway → AWS Lambda → Privacy Guard → Titan Embedding → CockroachDB Retrieval → Validated Evidence → Bedrock Investigator → Work Notes
The resulting analysis is written back into readable ServiceNow Work Notes.
Supporting evidence can include:
- historical incident number
- service
- semantic similarity
- verified historical root cause
- verified historical resolution
This keeps the recommendation inside the workflow engineers already use instead of forcing them to switch to a separate AI application.
The ServiceNow integration remains independent of the public Streamlit access gate.
No Streamlit session, judge access code, or browser state is required for the ServiceNow analysis path.
Agentic Incident Command Center
I also built a public Streamlit interface for exploring the system outside ServiceNow.
The Agentic Incident Command Center provides:
- reviewed synthetic investigation scenarios
- custom incident investigation
- a dedicated Privacy Guard scenario containing synthetic personal data
- visible Privacy Guard status and redaction metadata
- semantic similarity results
- grounded Amazon Bedrock recommendations
- structured supporting historical incidents
- an Operational Memory Graph
- real execution telemetry
- privacy-processing timing
- vector-retrieval timing
- Bedrock inference timing
- backend processing timing
- verified architectural security controls
- sanitized error handling
- bounded transient retries
The Operational Memory Graph intentionally shows only real retrieval relationships.
The current active incident appears in the center, surrounded only by the resolved historical memories actually returned by CockroachDB.
No artificial graph relationships are generated by the model.
Secure public demo execution
The Command Center itself remains publicly accessible and browseable.
However, triggering the live AWS / Bedrock investigation path requires a dedicated high-entropy Judge Access Code.
This protects the live cloud-backed execution path from uncontrolled automated use and unnecessary cloud cost while still allowing judges to inspect the application publicly.
The access control is intentionally isolated to the Streamlit demo layer.
The access code is never forwarded to:
- API Gateway
- AWS Lambda
- Amazon Bedrock
- CockroachDB
- ServiceNow
The first implementation used the concept of a short PIN.
A security review correctly identified that a short public PIN without shared server-side throttling would be vulnerable to online brute-force guessing.
Rather than presenting a per-session retry counter as sufficient protection, the final design rejects short PIN-style credentials entirely and requires a cryptographically generated, high-entropy access code.
This is intentionally a hackathon demo and cost-control boundary, not a replacement for enterprise identity management.
Security and production readiness
Security was treated as part of the architecture rather than an additional UI feature.
The current design includes:
- deterministic privacy redaction before configured direct identifiers reach embeddings
- privacy-aware sanitization before resolved incidents become durable CockroachDB memory
- secondary Bedrock Privacy Guard review using sanitized input only
- stop-before-AI behavior when privacy review requires further inspection
- retrieved historical evidence sanitized again before Investigator use
- active incidents excluded from trusted memory until resolution
- application-controlled memory lifecycle
- fixed application-owned vector retrieval
- strict memory scope enforcement
- no model-generated SQL
- no model-controlled retrieval count
- no model-selected supporting incident IDs
- Managed MCP operations constrained by application code
- GitHub OIDC and AWS STS temporary credentials
- no long-lived AWS access keys required by GitHub automation
- AWS Secrets Manager for integration credentials
- least-privilege IAM boundaries
- strict request validation
- vector-dimension validation
- evidence validation
- sanitized logs and frontend errors
- bounded automatic retry behavior
- API usage-plan protection for the ServiceNow route
- CI with Ruff and the full automated pytest suite
- high-entropy protection for public live demo execution
The system is therefore intentionally production-minded rather than fully autonomous.
The AI helps reason over evidence.
The application remains responsible for security boundaries, privacy enforcement, retrieval, scope, evidence selection, memory lifecycle, and write operations.
Challenges we ran into
One of the biggest design challenges was preventing the language model from becoming the source of truth.
A simple implementation could allow an LLM to decide what historical information to search for, what database query to execute, which incident looked relevant, and what information should be saved afterward.
That would be flexible, but difficult to trust in an operational incident-response environment.
Instead, retrieval remains deterministic and application-controlled.
CockroachDB returns evidence through a constrained vector-retrieval path, application code validates it, and Bedrock reasons only over the resulting evidence set.
Another major challenge appeared when persistent memory was introduced.
Operational incidents may contain names, phone numbers, email addresses, or other identifiers.
If those values are embedded first and sanitized later, the privacy boundary has already failed because sensitive information may already exist inside durable semantic memory.
The solution was to move privacy enforcement in front of both vector processing and persistent-memory creation.
Deterministic application code performs the actual redaction.
A narrowly scoped Privacy Guard agent can then review only the sanitized representation.
This means the model responsible for checking the privacy boundary does not need access to the values that were removed.
Another challenge was safely connecting multiple trust boundaries:
ServiceNow → API Gateway → Lambda → Privacy Guard → Titan → CockroachDB Managed MCP → Bedrock
Each boundary needed a clear responsibility.
Credentials had to remain outside source code.
Active incidents could not automatically become trusted memory.
The LLM could not control SQL or memory scope.
Historical evidence needed provenance.
Privacy needed to be enforced before AI processing and persistence.
The public demo introduced another security problem.
A short access PIN would have been easy to explain to judges but weak against automated guessing.
A security review identified that weakness before the final submission.
The design was changed to require a cryptographically generated high-entropy judge access code rather than pretending a client-side retry counter was a strong security control.
That review process reinforced an important principle behind the project:
automation should save engineering time without lowering the guard around data, identity, infrastructure, or operational memory.
Accomplishments that we're proud of
The current version includes:
- 50 verified resolved operational memories
- 60 reviewed synthetic ServiceNow incidents
- 10 active investigation scenarios
- semantic vector retrieval from CockroachDB
- CockroachDB Distributed Vector Indexing
- CockroachDB Cloud Managed MCP integration
- deterministic pre-AI Privacy Guard
- secondary Bedrock Privacy Guard reviewer
- sanitized-input-only separation of duties
- privacy-aware resolved-memory ingestion before embeddings and persistence
- evidence revalidation before Investigator use
- grounded Amazon Bedrock recommendations
- real execution telemetry
- live AWS backend deployment
- working ServiceNow integration
- public Streamlit Command Center
- Operational Memory Graph
- high-entropy judge access protection for live cloud execution
- GitHub Actions automation
- GitHub OIDC → AWS STS temporary credentials
- no long-lived AWS access key required for GitHub automation
- 256 automated tests
- 91.95% automated test coverage
What I am most proud of is that the project did not become "AI with unrestricted access to everything."
The design deliberately keeps deterministic software responsible for the boundaries where determinism matters.
AI is used where reasoning adds value.
Application code remains responsible for:
- privacy enforcement
- memory lifecycle
- provenance
- scope
- persistence
- retrieval
- validation
- evidence selection
- infrastructure access
That combination provides faster incident investigation without giving up the controls required to trust the result.
What we learned
The most important lesson was that useful agentic memory is not simply "more context."
Reliable operational memory needs:
- provenance
- lifecycle rules
- trusted write paths
- deterministic retrieval boundaries
- privacy controls
- evidence validation
- clear separation between unresolved hypotheses and verified historical outcomes
An active incident may contain theories.
A resolved incident contains knowledge that has passed through the operational process.
Those two things should not have the same trust level.
Another major lesson was that persistent agentic memory creates a different security question from normal generative AI:
It is not only "what can the model see?"
It is also:
"What are we willing to remember?"
Protecting the final LLM prompt is not enough if sensitive data has already been converted into an embedding and persisted.
That realization moved the privacy boundary earlier in the architecture — before vectorization and before durable memory creation.
We also learned that AI should not automatically become the control plane just because it can reason about the system.
CockroachDB retrieval, SQL selection, scope, supporting evidence, and memory writes remain application-owned.
The models operate inside those boundaries rather than defining them.
Finally, security review itself became part of the engineering process.
The public demo access mechanism evolved from a simple PIN concept into a high-entropy access control after reviewing its brute-force characteristics.
That reflects the same philosophy used throughout the project:
move quickly, automate aggressively, but keep the guard up around data and infrastructure.
What's next for Agentic Incident Memory
The current release focuses on evidence-grounded incident investigation and safe persistent operational memory.
A future closed-loop version could extend the same architecture from recommendation toward controlled remediation.
For example:
CI/CD failure → incident creation → log collection → memory retrieval → remediation proposal → human approval → pull request → CI validation → deployment → verified resolution → new operational memory
The important part is that the future system would preserve the same trust model.
AI could propose actions, but high-impact operations would remain controlled by deterministic policy and human approval.
Successful remediation could then become new operational memory only after the outcome had been validated.
The same architecture could eventually support:
- proactive detection of recurring failures
- automated correlation of new failures with previous incidents
- early remediation guidance
- GitHub Actions failure analysis
- application-log investigation
- controlled self-health workflows
- human-approved remediation
- continuously improving operational memory
For this hackathon release, the focus is deliberately narrower:
make persistent agentic memory useful, explainable, privacy-aware, secure, and trustworthy enough to support real operational investigation.
The core idea remains simple:
Remember what worked. Trust what was verified.
Built With
- AWS
- AWS Lambda
- Amazon API Gateway
- Amazon Bedrock
- Amazon Titan Text Embeddings V2
- AWS Secrets Manager
- AWS IAM
- AWS STS
- AWS SAM
- Amazon CloudWatch
- CockroachDB
- CockroachDB Distributed Vector Indexing
- CockroachDB Cloud Managed MCP
- ServiceNow
- GitHub Actions
- GitHub OIDC
- Streamlit
- Plotly
- Python
- pytest
- Ruff
- semantic search
- vector search
- agentic AI
- retrieval-augmented generation
- operational memory
- incident management
- privacy guard
- least privilege
- serverless
Built With
- actions
- agentic
- ai
- amazon
- amazon-web-services
- api
- bedrock
- cloud
- cockroachdb
- gateway
- github
- iam
- lambda
- mcp
- oidc
- pytest
- python
- rag
- sam
- secrets
- semantic
- servicenow
- streamlit
- titan
- vector
Log in or sign up for Devpost to join the conversation.