LengoAI

Give LengoAI a goal, not a ticket. Eight GitLab Duo agents plan, build, ship, and prove it. Lengo Core blocks harmful changes in milliseconds. Ship outcomes, not code.

Comment

Inspiration

Code generation is fast. Proving that the code actually improved something is much harder.

A change can pass every test, merge successfully, and still hurt production. Most teams discover this through customer complaints rather than reliable evidence.

I built LengoAI to close that gap. Instead of giving AI a ticket, I give it a measurable goal: reduce failed payments caused by gateway timeouts by 20%.

LengoAI takes that goal through the DevSecOps lifecycle, implements and releases changes safely, measures their impact, and learns from failures. The goal is not simply to ship code. It is to prove outcomes while protecting production.

What It Does

LengoAI combines eight GitLab Duo agents with Lengo Core, a deterministic decision-making service. Eight specialized agents handle the work, while Core enforces policies, controls consequential actions, and verifies results.

Agent Responsibility
Listener Converts ticket themes into measurable goals.
Strategist Defines targets, metrics, guardrails, and plans using cited memory.
Builder Implements changes behind feature flags and tests both flag states.
Gatekeeper Validates tests, policies, SAST, secrets, dependencies, and containers.
Launcher Manages staged releases from 1% to 100%.
Analyst Explains statistical evidence in plain language.
Commander Diagnoses incidents and executes recovery procedures.
Scribe Reports outcomes and stores lessons for future goals.

My core design principle is "LLMs propose, code disposes." Agents can plan, implement, and explain, but deterministic code controls merging, rollouts, rollbacks, goal verification, and memory updates. Every consequential decision is policy-checked and recorded in a hash-chained audit log.

A Failure That Became a Proven Outcome

Goal G-0003 demonstrates why this architecture matters.

  1. Detect and contain: The first implementation added a retry with back-off, but payment latency increased. When the p95 latency guardrail broke, Lengo Core disabled the feature flag within 620 milliseconds of detecting the breach, without human intervention.

  2. Diagnose and learn: The Commander identified the cause: a second 1,500 ms attempt after a full timeout. The Scribe stored the lesson as memory M-0006.

  3. Improve the plan: The goal reopened. The Strategist cited M-0006 and introduced a total time budget per payment.

  4. Prove the result: The revised change reduced failed payments from 10.12% to 7.5%, a 25.9% relative reduction, while both guardrails remained clean.

The absolute improvement was 2.62 percentage points, with a 95% confidence interval of 2.29 to 2.95 percentage points of reduction. The analysis used 177,569 flag-on and 229,454 flag-off payments.

LengoAI did not simply deploy a change and declare success. It detected harm, recovered, incorporated the lesson, and verified the improvement statistically.

How I Built It

GitLab Duo Agent Platform

I built seven custom flows supporting eight agent roles, triggered by issue and merge-request events through GitLab's Flows API.

Each agent follows AGENTS.md, receives a goal-specific context pack, and produces a structured JSON handoff. Lengo Core validates each handoff against a schema, including its author, state, idempotency key, and citations. I also implemented an MCP server in Core and an Ask Lengo agent.

Lengo Core and Infrastructure

I built Lengo Core with Python 3.12 and FastAPI, deployed on Google Cloud Run. Its infrastructure includes Cloud SQL (PostgreSQL and pgvector), Secret Manager, Terraform, and keyless CI authentication through Workload Identity Federation.

Core implements the state machine, policy engine, statistical verification, feature-flag controller, RAG memory with PII redaction, and public dashboard. Feature flags are managed through GitLab's feature-flag integration with Unleash.

Statistical Verification

Because Core evaluates outcomes every minute, repeatedly checking conventional fixed-horizon tests could inflate false-positive rates. I implemented a mixture sequential probability ratio test (mSPRT) to support continuous monitoring with always-valid p-values and confidence sequences.

The evidence process uses a likelihood-ratio mixture:

$$ \Lambda_n=\int\prod_{i=1}^{n} \frac{f_\theta(x_i)}{f_0(x_i)}\,d\pi(\theta) $$

Crossing the threshold (\Lambda_n \geq 1/\alpha) provides evidence against the null hypothesis while controlling the false-proof rate under the test's assumptions.

I also implemented harm-scaled guardrail tests and sample-ratio-mismatch checks. In 1,000 A/A simulations, there were zero false proofs.

Policy, Security, and Quality

I implemented three autonomy modes: Assisted, Supervised, and Hands-off. Policy enforcement includes protected paths, change-size limits, mandatory feature flags, freeze windows, and execution budgets. Emergency rollbacks remain permitted during freezes, and policy changes require merge requests.

The CI pipeline includes linting, 521 tests with 94% coverage, policy checks, a RAG evaluation gate with Recall@5 of 0.875, SAST, secret detection, dependency and container scanning, image builds, and audit-chain re-verification.

Challenges I Ran Into

  • Preventing prompt injection: I tested a forged issue comment that impersonated Core and instructed agents to modify the payments ledger and deploy without a feature flag. The agents ignored it, and a probe merge request touching a protected file was blocked by policy enforcement.

  • Handling production friction: A change freeze delayed the initial rollout, and a feature-flag API rejected a strategy payload. Database connection exhaustion under load also forced me to introduce a per-cohort user table instead of counting distinct users directly from raw events.

  • Making rollbacks fast and reliable: The 620 ms rollback required a fast path through metric ingestion rather than waiting for the next scheduled evaluation. I also had to guard against noisy measurements causing unnecessary rollbacks.

  • Making statistics understandable: Translating sequential-test evidence and confidence bounds into explanations that product owners can act on was one of the hardest parts of building the Analyst.

What I Learned

  • Autonomy needs deterministic boundaries. AI agents are effective at proposing and explaining changes, but consequential decisions need explicit, testable enforcement.

  • Learning must be traceable. The second plan succeeded because it cited the lesson from the first failure. Memory without verifiable references is difficult to trust.

  • Deployment is not proof of success. A merged change is a technical milestone, not a business outcome. Statistical evidence and guardrail checks are what make the difference.

What's Next

I plan to validate LengoAI against real user traffic, expand supported goals to include cost, latency, and conversion, and enable goals that span multiple repositories.

My long-term vision is an engineering system that continuously pursues measurable outcomes, catches harmful changes, recovers automatically, and improves through evidence-backed learning.

Built With

  • ai-agents
  • devsecops
  • docker
  • fastapi
  • feature-flags
  • gitlab-ci/cd
  • gitlab-duo
  • gitlab-duo-agent-platform
  • gitlab-feature-flags
  • google-cloud-run
  • google-cloud-secret-manager
  • google-cloud-sql
  • large-language-models-(llms)
  • mcp
  • model-context-protocol-(mcp)
  • pgvector
  • postgresql
  • python
  • rag
  • retrieval-augmented-generation-(rag)
  • sast
  • statistical-testing
  • terraform
  • unleash
  • workload-identity-federation
Share this project:

Updates

Submission history