Vulcan

Accuracy Metric
Database tables
error occurs
agent checks existing error examples and takes help from LLM and self-heal
error occurs and unable to self-heal
new incident logged
Human intervenes to resolve the issue
order placed! ( the error is fixed by the agent)
investigating a standalone error

Inspiration

Every on-call engineer has fixed the same bug twice — once at 2am under pressure, and again months later, because nobody remembered the first fix. Incident response tools are great at alerting, but terrible at remembering. Runbooks go stale, tribal knowledge lives in someone's head or a random Slack thread, and teams re-diagnose problems they've already solved. We wanted to build a system that treats every resolved incident as a permanent addition to institutional memory, not a one-off.

What it does

Vulcan is a self-healing incident manager. When a new incident comes in, it embeds the symptoms and searches CockroachDB's vector index for similar past incidents using vector similarity. For a high-confidence match, Vulcan retrieves a previously validated resolution and allows the AI agent to implement and validate the fix automatically. If the match isn't confident enough, it routes the incident to a human, who investigates, applies the fix, and records it. That resolution — along with its embedding — gets written back into CockroachDB, so the next time a similar incident occurs, Vulcan recognizes it and can act on its own.

How we built it

Vulcan is built as a Python/Flask backend hosted on Render, with the agent pipeline also designed to run as an AWS Lambda function for serverless execution. We use psycopg connection pooling to efficiently reuse database connections across requests.

When an incident occurs, the agent processes it through five stages:

Embed — OpenAI's text-embedding-3-small converts the incident into a 1536-dimensional vector. Retrieve — CockroachDB's vector index performs cosine-similarity search to find related incidents. Load Skills — The agent loads the relevant CockroachDB SKILL.md best-practice document from the official Skills Library. Reason — GPT-4o-mini analyzes the symptoms, similar incidents, and skill context to determine the root cause and appropriate fix. AWS Bedrock is also supported as an alternative provider. Heal — The agent updates a boolean flag in CockroachDB's heal_flags table, enabling the fixed code path.

CockroachDB serves as our single data layer. It stores incident embeddings and resolution history, powers vector similarity search, manages runtime healing flags, and stores the demo application's business data. This allowed us to avoid introducing a separate vector database or additional infrastructure.

Our ShopEasy demo application checks the healing flag on each request. When an issue is identified and healed, it automatically switches to the fixed code path. Errors are reported to Vulcan in the background, so the healing process happens without disrupting the user experience.

The frontend is a server-rendered HTML interface with vanilla JavaScript. The incident manager provides investigation, pending-review, and history views, along with resolution statistics such as auto-resolved versus manually resolved incidents.

Challenges we ran into

We originally built on AWS Bedrock but hit access issues mid-hackathon. Switching to OpenAI was contained to the reasoning layer because we'd kept it decoupled — vector search and healing logic didn't change. Corporate proxy issues with Python's SSL trust store cost us extra debugging time.

After the switch, we faced Vector dimension mismatches between providers (1024 vs 1536) that caused silent write failures until we traced the issue. To confirm the CockroachDB skills library was actually being used, we added instrumentation — the API response shows which skill was loaded, proving it's fed to the LLM at reasoning time.

On the CockroachDB side, we initially had some confusion around organizations, clusters, and IAM. The CockroachDB Assistant helped us quickly identify and resolve the issue, allowing us to get back to building without significant database-related delays.

Accomplishments that we're proud of

The moment that made Vulcan feel real was watching the learning loop close: an unfamiliar incident gets escalated to a human, fixed and recorded — and when the same incident occurs again, Vulcan can confidently auto-resolve it. That demonstrates our core thesis working end to end.

Our retrieval pipeline achieved 95.0% Precision@1 and 99.5% Precision@3 across 200 test incidents, while root-cause identification reached 96.5% accuracy. Our overall combined score was 95.75%.

We also handle deduplication, so repeated occurrences of the same error don't flood the system. The dashboard provides live visibility into total incidents, auto-resolved incidents, open incidents, and affected services.

From the CockroachDB side, we successfully integrated 3 of the 4 available CockroachDB tools, using CockroachDB as the central system for business data, vector search, healing flags, and incident memory — without needing a separate vector database, Redis, or additional infrastructure.

What we learned

We learned that CockroachDB's native vector search can eliminate the need for a separate vector database — a single cluster can handle both transactional data and semantic similarity search.

We also found that storing self-healing state as database flags, rather than modifying files or maintaining patches, makes the system simpler and easier to deploy across different environments.

One of our biggest takeaways was the value of incident memory. The system may require human intervention when it encounters a new incident, but once the resolution is recorded, future occurrences can be recognized and automatically resolved.

Finally, designing our AI layer to be provider-agnostic from the beginning proved valuable when we had to switch from AWS Bedrock to OpenAI during the hackathon. Because the reasoning layer was decoupled from the rest of the pipeline, we could change providers without redesigning our vector search or healing logic.

What's next for Vulcan

Our next step is to integrate Vulcan with Prometheus and Grafana, allowing real monitoring alerts to automatically trigger incident analysis and healing.

We also plan to add post-fix health checks so Vulcan can validate that a repair actually resolved the issue before marking the incident as healed. Human feedback will be used to calibrate confidence thresholds — operator overrides can become training signals for improving future decisions.

On the AI side, we want to expand our use of AWS Bedrock to make Vulcan more flexible and model-agnostic, allowing us to choose between different foundation models based on the incident and deployment requirements.

Finally, we want to leverage CockroachDB's distributed architecture for multi-region incident memory, allowing globally distributed teams and services to share a single, resilient knowledge base.