Magitly

Magitly
Agents Activity Overview
Agents Pipeline and Status
Auto-commit by Refactor Agent
Project Manager Agent Interface

Inspiration

Every developer on our team has lived this exact frustration.

You push code at night. Something breaks. By morning, three people have been pulled into a thread trying to figure out what happened, why, and who should fix it. A new developer joins and spends their first two weeks asking the same questions senior devs have answered a hundred times before. A PM sends a Slack message asking what shipped this sprint — and five engineers get context-switched out of deep work to answer it.

AI coding tools exist. We all use them. But every tool we tried — Cursor, Copilot, Devin — makes individual developers faster. None of them know your team. None of them remember what broke last month, who owns which module, or why a particular architectural decision was made six months ago.

The insight that started Magitly was simple: AI has been applied at the individual level, not the team level. The coordination overhead that kills engineering velocity — broken pushes, slow onboarding, PM interruptions, lost context — none of that lives in a single developer's editor. It lives in the space between developers. And nothing was solving it.

We wanted to build the AI that lives in that space — not another autocomplete, but a team member.

What it does

Magitly is a self-healing CI/CD pipeline and team intelligence layer for engineering teams, built as a society of specialized AI agents that share a single persistent knowledge graph. Connect a repo, and two things start happening immediately: every push gets autonomously reviewed and fixed, and the entire team gains a project brain they can ask anything.

The Self-Healing Pipeline

Every code push triggers a GitHub webhook that fires a chain of five agents, each with one job and no overlap:

Secret Scanner (deterministic, not LLM-based) — parses the raw diff into per-file added-line content, strips out all diff plumbing (headers, hunk markers, removed lines, context lines), and runs detect-secrets against only the real new code. Findings are mapped back to the actual file path and real line number in the new file — not the temp file used internally for scanning. No API key, token, or credential ever reaches an LLM.
Planner Agent — triages the diff: is there actually a problem here, or is this push clean?
Reviewer Agent — diagnoses the root cause and produces a structured, step-by-step fix plan (not code — instructions for the next agent).
Refactor Agent — writes the actual patch, following the fix plan exactly, touching only what's necessary.
Tester Agent — validates the fix using structured LLM reasoning: it writes targeted test cases, reasons through at least three scenarios (happy path, edge case, and the specific failure the patch was meant to resolve), and returns a structured pass/fail verdict.

If tests pass, the patch is committed automatically to the branch. If they fail, the Refactor Agent retries — with the entire failure reason and reasoning output from the previous attempt attached — up to 3 times. If all 3 fail, the developer who pushed gets a Slack DM with a complete, human-readable summary of every attempt and why the pipeline gave up. Nothing is retried blindly; every retry is informed by exactly what went wrong last time.

The Project Manager Agent

A persistent project brain that ingests the entire codebase on first connect — file contents, module structure, and per-file commit history pulled from the GitHub REST API — and builds a knowledge graph plus a RAG store. Every file gets attributed to its most frequent recent committer, so ownership isn't guessed, it's derived from real git history. PM Agent then answers any question about the project in plain English, grounded in that graph:

A Slack bot where PMs and engineering managers can ask "what shipped this sprint?" or "who owns the payments module?" without pinging a single developer
An onboarding flow where new developers get the entire project explained to them — architecture, modules, recent decisions, who to ask about what — without burning a senior dev's afternoon
An in-app chat interface where existing developers ask questions and get context-aware answers while they work, complete with file citations so every answer is traceable back to real source

Every agent in the pipeline writes its findings back into the PM Agent's knowledge graph after every run. The system doesn't just execute — it accumulates. A question asked after ten pipeline runs gets a richer, more grounded answer than the same question asked on day one.

How we built it

Agent orchestration. We used LangGraph to build the pipeline as a compiled StateGraph. Each agent is a node that reads from and writes to a shared PipelineState dataclass — nothing is passed directly between agents, everything flows through this single object. The retry loop (Tester → Refactor → Tester) is expressed as a conditional edge with a hard cap at 3 attempts before escalating. We used Annotated[list[...], operator.add] on every accumulating state field — patches, test_results, events, diagnoses, secret_findings — so LangGraph concatenates partial updates across retries instead of overwriting them. Getting this wrong silently destroys retry history and makes the escalation logic produce empty, useless summaries — a failure mode that doesn't throw an error, it just quietly gives you the wrong answer.

LLM layer. All agents run on Qwen Cloud via the OpenAI-compatible DashScope endpoint. We used qwen3.7-max for code reasoning (Reviewer, Refactor) and qwen3.7-plus for triage and coordination (Planner, PM Agent). Every agent's LLM call goes through a defense-in-depth output parser: strip markdown fences, extract the JSON object via regex, parse, construct the Pydantic model explicitly from the raw dict rather than trusting JsonOutputParser's type coercion, and fall back to a safe failure state — never a crash — if anything goes wrong.

Knowledge graph and RAG. The PM Agent uses a dual-store architecture: an in-memory fallback for development, and a Postgres + pgvector store on Alibaba Cloud RDS for production, switched transparently by a single db.is_configured check. The codebase is chunked with overlap and embedded using Qwen's text-embedding-v4 model (1536-dimensional vectors). Semantic search uses pgvector's cosine similarity; a PostgreSQL websearch_to_tsquery text search acts as a fallback. Beyond file and module nodes, the graph stores contributor nodes and "owns" edges — built by fetching per-file commit history from the GitHub REST API and attributing ownership to the most frequent recent committer, weighted by commit count. This is what lets PM Agent answer "who worked on this" with a real, traceable answer instead of a shrug.

Integrations. GitHub MCP and Slack MCP both run as stdio subprocesses spun up via npx inside FastAPI's async lifespan, bridged into LangGraph-compatible tools via langchain-mcp-adapters. Each integration is independently feature-flagged so the app can start cleanly with only the credentials that are actually configured. A dedicated webhook receiver verifies GitHub's X-Hub-Signature-256 HMAC signature against the configured secret, parses push payloads (correctly distinguishing real pushes from branch deletions, tag pushes, and pings), fetches the full diff via GitHub MCP using the commit SHA, and fires the pipeline as a background task — returning a response to GitHub in well under its timeout window while the actual agent work continues asynchronously.

Stack:

Python · FastAPI · LangGraph · Qwen Cloud (qwen3.7-max, qwen3.7-plus, text-embedding-v4)
GitHub MCP · Slack MCP · Postgres + pgvector (Alibaba Cloud RDS) · Next.js

Challenges we ran into

LangGraph state merge semantics. Fields written by more than one node across the retry loop — patches, test_results, events — silently get overwritten instead of appended unless explicitly declared Annotated[list[...], operator.add]. This isn't loudly documented and cost us real debugging time before we traced why our escalation summaries were coming back with only the last attempt instead of the full retry history.

LLM JSON output reliability. Qwen models occasionally wrap JSON in markdown fences, prepend reasoning text in <think>...</think> blocks, or produce structurally valid-but-mismatched JSON. We built a multi-stage parser used identically across every agent — strip, extract, parse, validate, construct the Pydantic model explicitly, and fall back gracefully — rather than letting a single malformed response crash the pipeline mid-run.

Diff parsing for the Secret Scanner. Our first version scanned the entire raw diff blob, which meant it flagged secrets in removed lines, matched findings against the wrong file (the scanner's own temp file path instead of the real repo path), and reported the wrong line numbers. We rewrote the scanner to parse unified diffs properly — isolating only added lines per real file, mapping temp-file line numbers back to real line numbers in the new file, and scanning each file's added content in isolation so findings are accurate enough to act on directly.

GitHub webhook signature and routing debugging. Getting the webhook working end-to-end meant working through a full stack of failure modes in sequence: a 405 that turned out to be an Nginx reverse-proxy misconfiguration in front of FastAPI, then a 401 that traced back to a webhook secret mismatch between GitHub's stored secret and our .env. Each layer — DNS/proxy, transport, application, signature — had to be verified independently using GitHub's Recent Deliveries log before we could confirm the pipeline was actually firing.

Contributor and ownership data. File content ingestion alone tells PM Agent what a file does, not who worked on it. We had to add a separate ingestion pass that fetches per-file commit history from the GitHub REST API, attributes ownership by commit frequency, and writes it into the graph as first-class contributor nodes and owns edges — then made sure that data actually gets rendered into the prompt context sent to Qwen, since fetching it and using it turned out to be two separate bugs.

MCP subprocess lifecycle. Running GitHub MCP and Slack MCP as npx subprocesses inside FastAPI's async lifespan required careful startup ordering — tools had to be fully loaded before any request handler could call them — and clean shutdown to avoid orphaned Node.js processes. Independently feature-flagging each integration meant the app could keep running in a partially-configured state during development instead of refusing to start.

Alibaba Cloud provisioning. Getting RDS PostgreSQL with pgvector live in an international region meant navigating account verification, region and engine-version availability constraints (PostgreSQL 18 had no available instance types in some regions; PostgreSQL 16 in Frankfurt worked), and installing pgvector through the Plugin Marketplace rather than a plain CREATE EXTENSION command.

Accomplishments that we're proud of

The retry loop is genuine agent negotiation, not blind repetition. The Tester Agent's full failure reason and reasoning trace are fed back into the Refactor Agent's context on every retry. The Reviewer's original diagnosis survives in shared state, so by attempt three the Refactor Agent still knows exactly what the root problem was and what's already been tried and failed. The agents are coordinating across turns, not just re-running the same prompt.

A knowledge graph that answers "who," not just "what." Most RAG-over-code systems stop at file contents. Magitly's PM Agent ingests commit history alongside code, builds real ownership edges weighted by commit frequency, and can answer "who owns this module" or "who worked on this feature" with a traceable, graph-backed answer — not a guess.

A knowledge graph that doesn't need a graph database to develop against. The dual-store pattern — in-memory for development, Postgres + pgvector for production, switched by a single flag — means the entire system runs end-to-end with zero external dependencies locally. Any teammate can clone the repo and run the full pipeline immediately, no database provisioning required to start contributing.

Security-first by construction, not by policy. The Secret Scanner is the one component in the entire pipeline with no LLM in it at all, and it runs first, unconditionally, before any model ever sees a diff. That was a deliberate architectural choice: the most sensitive check in the system doesn't get to be probabilistic.

A pipeline that survives real infrastructure failure modes. Getting the webhook live meant debugging real proxy misconfiguration, signature mismatches, and payload parsing edge cases — and the system now handles all of it: ping events, branch deletions, tag pushes, and malformed payloads are all distinguished cleanly rather than crashing the receiver.

What we learned

Building a multi-agent system that is genuinely more capable than a single agent is harder than it sounds. Most "multi-agent" systems are really just pipelines with a router — each agent runs sequentially and hands off to the next with no real memory of what came before. What makes a system an actual society is shared memory, back-and-forth negotiation, and agents whose behavior changes based on what other agents have already done.

The Tester → Refactor retry loop, informed by real failure context, is the moment Magitly crosses that line. The Reviewer's diagnosis surviving in shared state so the Refactor Agent on attempt three still knows the original problem — that's the property that makes this a society rather than a chain.

We also learned that LLM reliability in production is an engineering problem, not a prompting problem. The perfect prompt still breaks if you don't build robust JSON parsing, structured output validation, and graceful fallback paths — a single unexpected <think> tag or stray sentence before a JSON object will take down a naive pipeline. Defense in depth — strip, extract, parse, validate, fall back — is now a pattern we apply to every LLM output we handle, everywhere.

We learned that data you fetch but never render is functionally the same as data you never fetched. Our contributor ingestion worked correctly weeks before "who worked on this" questions actually got answered correctly — because the ownership edges were being pulled from the graph but never rendered into the prompt context sent to the LLM. The bug wasn't in ingestion; it was in the last mile between memory and language.

Finally, the right abstraction boundary matters enormously. Keeping PM Agent as the single source of truth every other agent calls into — rather than letting each agent build its own local context — is what keeps the system coherent as it grows. Hub-and-spoke with a persistent brain at the center is the right architecture for a multi-agent team product.

What's next for Magitly

Coding Agent. PM Agent currently answers questions and explains code. The next step is a Coding Agent that writes code alongside a developer in real time — pulling context from the knowledge graph, understanding existing conventions, and suggesting completions consistent with the actual codebase rather than generic boilerplate. This is what takes Magitly from "team intelligence" to "AI pair programmer for the whole team."

Sandbox execution. The Tester Agent currently validates patches through structured LLM reasoning. The next evolution adds Qwen's code_interpreter tool — actually executing the patched code and generated tests in a sandbox rather than reasoning about them statically, making test results empirically verifiable rather than judgment-based.

Persistent knowledge graph at scale. With Alibaba Cloud RDS + pgvector fully provisioned in production, PM Agent's memory persists across restarts and keeps accumulating real project history indefinitely. The first team that runs Magitly for a month will have a PM Agent that knows more about their codebase's history than any individual engineer does.

Email escalation. Slack DM escalation is live today. Adding Resend for email fallback closes the gap for teams where the developer who pushed isn't actively watching Slack.

Production hardening. Token budget management across long pipeline runs, automatic retry on LLM parse failure instead of a hard stop, rate limiting on the public API surface, and full telemetry so teams can see exactly what every agent did and why — these are what turn a hackathon build into something a real engineering team trusts with their production branch.