Inspiration

The worst bugs we've shipped weren't inside a function. They were in the wiring between things. A queue that never sent failed messages to its dead-letter queue. A retry setting that quietly dropped work. A Lambda missing one permission. Lint passed, unit tests passed, tofu validate passed, and the bug still went to production.

When a pipeline does catch something like this, you usually get a red X and a very long log. Coding agents have it worse. They can only check their changes against the same unit tests, and when CI fails they're guessing from text. We wanted people and agents to be able to look at the broken environment itself, not a description of it.

What it does

Macarena gives every commit its own throwaway sandbox with the real pieces running: SQS, Lambda and DynamoDB in LocalStack, set up by the repository's own OpenTofu, plus the app and its integration tests. It returns one verdict for the commit.

When the verdict is FAIL, Macarena doesn't tear the sandbox down straight away. It holds it for a few minutes, asks it read-only questions (queue attributes, table contents, deployed functions), and puts the answers in the job log next to the failed test and the log lines written while it ran.

On GitLab, that turns into a supervised loop:

A merge request "tidies up" the dead-letter setup on an SQS queue. It looks reasonable, and lint and unit tests pass.
The macarena:validate job runs Macarena inside the pipeline, and the dead-letter test fails. The log shows why: the orders queue has no RedrivePolicy at all, and the bad message is still sitting in it.
The developer mentions our macarena-fix flow on the merge request. It's a GitLab Duo Agent Platform flow with two agents. The first reads Macarena's evidence, the diff and the code, and works out the cause. The second puts the redrive policy back, commits to the branch, and leaves a note saying what it saw, why it happened and what it changed. In our run, that took about a minute.
The new pipeline checks the fix in a fresh sandbox, and the verdict comes back PASS. The verdict is signed, and the job verifies the signature against the exact image it built.
A person reviews the result and merges. That's the only human step.

If the sandbox itself dies halfway through, Macarena marks the attempt as lost and runs it again, instead of blaming your code.

Who it's for: teams whose services depend on cloud wiring (queues, functions, tables, permissions), and the coding agents working in their repositories.

How we built it

Go, start to finish. A control plane on PostgreSQL with leases and fencing, a host agent that runs sandboxes, and a guest agent inside each one.
Sandboxes: a Docker provider for CI and laptops, Firecracker microVMs for real isolation, and EC2.
Infrastructure: LocalStack for the AWS services, and OpenTofu with an offline provider mirror.
Evidence: logs, JUnit and inspect results go to S3. Verdicts are signed as DSSE / in-toto attestations, with a local key or AWS KMS. Each validation gets one OpenTelemetry trace.
For agents: an MCP server exposes the same controlled inspect queries the pipeline uses.
Safety: secrets are sealed at rest, handed only to the host running the job, and redacted from every log. Sandbox network access is off by default, or limited to an allowlist of domains.
On GitLab: a .gitlab-ci.yml that runs lint, the test suite and Macarena itself inside the job, plus the macarena-fix flow in .gitlab/duo/flows/.

Everything is tested against real systems: Postgres, S3, a container registry and LocalStack. We didn't use mocks.

Lifecycle stages it touches

Create: the flow edits the code and commits the fix to the merge request's branch.
Verify: lint, test and macarena:validate run on every push. The verdict comes from real integration tests in a real sandbox, and the JUnit report shows up on the merge request.
Package: every pipeline builds the app and test-runner images and the OpenTofu bundle for that commit, and Macarena validates exactly what was built.
Secure: secrets are sealed, scoped and redacted, and sandboxes have no network access unless it's allowlisted.
Govern: a per-project policy decides what a sandbox may do, every verdict is signed and verified, and a person approves the merge. The agent never merges.

Challenges we ran into

Running Macarena inside a CI job meant a Docker daemon inside the job, the dev stack inside that, and sandboxes inside those. We fought overlay filesystems, cgroups and container networking at every layer before we got the first green run on GitLab's runners.

Those runners also caught a bug our laptops never would have: our tests assumed an arm64 machine, and GitLab's runners are x86_64.

The other hard part was telling platform failures apart from app failures. A killed sandbox or a slow LocalStack start shouldn't fail your merge request. We ended up treating "lost" as its own outcome, with its own retry.

Accomplishments that we're proud of

The whole loop runs for real on GitLab, and anyone can check it. Merge request !2 has the failing job, the mention, the flow's commit and note, and the green pipeline after it.

The flow's note quotes Macarena's evidence directly, including the queue attributes from the held sandbox. That was the whole point of the project: the agent fixed the bug by looking, not guessing.

GitLab Duo's own code review flagged the same regression on its own, which reassured us the bug was a realistic one and not something we'd made easy.

What we learned

The most useful thing to hand an agent isn't more log. It's the live environment, held open, with safe questions it can ask.

We also learned that a testing platform has to be honest about its own failures. If a crashed sandbox looks the same as a broken app, people learn to ignore red builds.

What's next for Macarena

Firecracker hosts on Google Cloud, so the sandbox is a hard isolation boundary in CI as well. The Docker provider we use in CI today is a development provider, not a security boundary.
More inspect queries for agents to ask.
Letting the flow start on its own when macarena:validate fails, and request a re-validation itself, so nobody has to mention it.