NoMeshOps — self-healing deployment agent

Automates Python deployments on EC2, diagnoses failures using rules, DynamoDB, and Bedrock, and verifies every fix before saving it for instant reuse.

Comment

Inspiration

Every developer has deployed to a fresh machine and watched it fail on something that has nothing to do with their code: a missing system library, a package that needs a compiler that isn't there, an app that installs cleanly and then crashes at startup. You SSH in, read the log, run one command, redeploy, and repeat. The failures are repetitive, but the fix depends on the machine's OS, architecture and Python version, so a generic script doesn't solve it. We wanted an agent that does that loop itself and only believes a fix once it has proven it works.

What it does

NoMeshOps is a self-healing deployment agent. You give it a target machine and a Python repo. It fingerprints the machine, deploys the project, and if the deploy fails it classifies the failure and fixes it. It tries three things in order: a deterministic rules table (sub-millisecond, no network), then a DynamoDB knowledge base of previously verified fixes, and only for genuinely new errors one capped Amazon Bedrock (Claude) call.

A fix only counts as verified when all of these hold: the install succeeds, every declared package imports, the app process starts, the health endpoint returns 200, and the process that answered is the one we just started rather than a stale one holding the port. Verified fixes are stored, so the same error resolves near-instantly the next time, even on a different platform through a family index that still passes the same verification gate.

It works from a CLI (nomeshops login, an aws-login-style device sign-in, then nomeshops deploy) and from a hosted web demo that deploys any public repo live and streams every step.

How we built it

  • Orchestration: FastAPI plus a LangGraph state machine: scan, fingerprint, deterministic check, deploy, verify, finalize, with the rules, knowledge base, Bedrock ladder looping back to retry (max 3 fix attempts, 1 model call per run).
  • AWS: ECS Express Mode hosts the server. Systems Manager Run Command does every interaction with a target machine (no SSH keys, no inbound ports). DynamoDB is the knowledge base, keyed by a hash of error type, package, version, OS, architecture and Python minor, with a family-index GSI. Bedrock (Claude, Converse API) generates fixes on a miss, behind a destructive-command deny-list. S3 holds a full audit record of every attempt. ECR, IAM and CloudWatch round it out.
  • Pluggable backends: the same code runs against local Docker containers, local files and the Anthropic API for cheap development, or against SSM, DynamoDB, S3 and Bedrock in production.
  • Web platform: email and password accounts (PBKDF2), a device-code sign-in shared with the CLI, a per-account quota on the live demo, and a repo host allow-list.

Challenges we ran into

  • Verification that can't lie. Early versions reported success when a stale process was still answering the health check. We added a responder-identity check (port ownership via /proc) so only the process we started can pass.
  • Real infrastructure disagreed with local testing. The demo's target list used Docker container names, which only work in local mode. Against real AWS, SSM needs actual EC2 instance IDs, so four of six targets would have failed on click. We found it by redeploying and checking, and fixed it.
  • An identity bug in our own login. The terminal sign-in and the real account system were two disconnected token stores, and the approval trusted a client-supplied name. A real nomeshops login token would not have worked anywhere. We unified them onto one store.
  • Alpine Linux broke three assumptions at once: no bash, py3-* package names instead of python3-*, and command -v with several names silently returning nothing under busybox.
  • Not learning the wrong lesson. Network blips were being stored as "fixes". We added a transient-failure classifier so they are retried and never learned.
  • AWS onboarding friction: the Bedrock first-use verification error, region-specific inference profiles, and aws login failing for a federated hackathon role.

Accomplishments that we're proud of

  • Every number comes from real runs on AWS: a Bedrock-assisted fix in about 42 s, a knowledge-base hit with zero model calls in about 34 s, a deterministic fix in about 58 s, and a full ECS Express Mode run in 29.6 s with a 45 ms knowledge-base lookup.
  • A fix learned on Debian transferred to an Azure-style Ubuntu target with zero additional model calls, because it was already verified once.
  • A formal accuracy and reliability pass over the system found and fixed six bugs. Every bug we hit now has an automated test (27 passing).
  • Treating Bedrock as a last resort is a cost decision as much as a design one, since it is the only service in the stack with no free tier.
  • We disclosed the system's limits instead of hiding them.

What we learned

  • Test against the real platform early. Every serious bug we found only appeared against real AWS, real Alpine, or a real login flow.
  • "The command exited 0" is not "it works". Verification should prove the thing you care about, down to which process is answering.
  • Runtime context is part of an error's identity: the same error text on Python 3.9 and 3.12 can need different fixes, so the signature has to encode it.
  • Deterministic first, probabilistic last. Most failures are already known, which makes the common path fast and free and keeps the model for what's actually new.
  • Honest scope notes build more trust than polish does.

What's next for NoMeshOps — self-healing deployment agent

  • Per-account target ownership. Signing in proves identity, not authorization over specific machines. Today any signed-in account can deploy to any instance the server can reach. Registering machines per account is the first thing to close.
  • Real machines for every cloud. Four of the six targets (GCP, Azure/DigitalOcean, Oracle/on-prem, Alpine) are simulated with local Docker containers. Next is reaching real hosts on those clouds, not only EC2 over SSM.
  • Beyond Python: Node, Go and containerized apps, with the same verify-before-trust loop.
  • Production-grade state: move accounts and sessions from a local JSON file into DynamoDB, and add per-user quotas and audit views.
  • Smarter learning: confidence scoring, fix expiry and re-verification so the knowledge base improves instead of just growing.

Built With

Share this project:

Updates

Submission history