Inspiration## Inspiration
On-call engineers face the same incidents over and over. A service overloads, a dependency goes down — and every time, someone has to diagnose it from scratch. Runbooks go stale, context is lost between shifts, and response times degrade under alert fatigue. We wanted an agent that actually remembers — one that gets smarter with every incident it handles.
What it does
Ops-Sentinel is an autonomous on-call agent with persistent incident memory. It continuously monitors a target service, detects anomalies, and — instead of starting from a blank slate — searches its own history of past incidents using semantic vector similarity. When a fault recurs, the agent retrieves the most similar past cases (with cosine similarity scores), feeds them to Qwen to reason over what happened before, then executes a remediation action and records the new incident with an embedding for future retrieval.
How we built it
- Agent loop: a Python/FastAPI service polls a demo service every 10 seconds, computing health and metrics.
- Memory: every incident (symptoms, metrics, diagnosis, action, outcome) is persisted in SQLite, with a 1024-dim embedding from Qwen text-embedding-v3.
- Semantic retrieval: incoming anomalies are embedded and matched against history by cosine similarity; the top matches with scores are surfaced on the live status page.
- MCP layer: incident memory is exposed as an MCP server with 4 tools (search_similar_incidents, record_incident, get_recent_incidents, get_stats) over FastMCP Streamable HTTP.
- Autonomous reasoning: Qwen (qwen3.6-flash) drives a diagnose → decide → act loop, calling the MCP tools autonomously via function-calling — no hardcoded decision tree.
- Deployment: three Docker containers (demo service, agent, MCP server) running live on Alibaba Cloud ECS.
Challenges we ran into
A local_address=127.0.0.1 binding on the HTTP client (a Windows-only workaround) silently broke service health probes inside Docker with EINVAL, generating thousands of false "service unreachable" incidents until we traced the root cause. The semantic search kept falling back to text matching because the embedding client wasn't initialized in the MCP server container. FastMCP list serialization and Unicode handling on an ASCII-locale container surfaced subtle encoding bugs that took focused debugging.
Accomplishments that we're proud of
A fully autonomous agent that reasons over its own memory via MCP, with no prompt-engineering tricks or hardcoded rules. Semantic retrieval with cosine similarity scores visible right in the UI — proof the memory is conceptual, not keyword-based (matches found with sim ≈ 0.94 and zero keyword overlap). Graceful degradation: if the MCP server is down it falls back to text search; if Qwen is down it falls back to rule-based decisions — the agent never crashes. And it's a real, deployed system on Alibaba Cloud, not a local toy demo.
What we learned
Distributed systems fail in subtle ways: the hardest bug was a one-line socket binding that worked locally but silently broke inside containers. We learned to add explicit diagnostic logging instead of silent fallbacks, to verify each layer (embedding client, MCP server, agent) independently, and that a clear, observable UI — showing exactly which tool was called and with what similarity score — is as important as the underlying logic for proving the system works.
What's next for Ops-Sentinel
- Human-in-the-loop "copilot" mode: agent proposes a fix and asks for approval before acting on critical incidents.
- Negative-experience memory: tag actions that made things worse so the agent learns what not to repeat.
- Proactive prediction: detect rising trends (e.g. memory growth) and remediate before an outage.
- Auto-generated post-mortem reports from the incident chain.
Log in or sign up for Devpost to join the conversation.