Legacy Lifeline

• 

  The rescue as a GitLab pipeline: scan, build, ABI gates, security scans, side-by-side test, package, rehearsal, manual fleet rollout.

• 

  Seven ABI gates, each one a lesson from the field. All pass on main.

• 

  PHP 7.3 on :80 and PHP 8.3 on :8080, same Apache binary on a device clone: every response matches.

• 

  A cleanup drops one linker flag. The symbol isolation gate catches 3,096 leaked OpenSSL symbols: the bug that crashed Apache.

• 

  The pipeline opens an incident. One comment hands it to the GitLab Duo flow; 86 seconds later it opens a fix MR.

• 

  Duo's MR: root cause, a one-line fix with a warning comment, and why it is safe on the device.

• 

  GitLab Duo Code Review caught a risky installer change by a human and blocked the merge until it was fixed.

• 

  Installer rehearsed on a clone: install, re-run, rollback, re-install. Fleet rollout waits for human approval.

• 

  The CI-built ARM module on the real ARMv7 box: install, re-run, rollback, re-install through the same installer. PHP 7.3.2 → 8.3.36.

Inspiration

I maintain Apache + mod_php 7.3 boxes installed in shops. In October 2026 an RCE was published for EOL PHP 7.3. The box vendor said updating PHP meant rebuilding the whole Linux image — "almost impossible". The boxes have a read-only rootfs, no compiler, no scp, and a noexec /tmp. I rescued one by hand: cross-compiled a single PHP 8.3 libphp.so and swapped it in. It took days, and the worst bug was invisible until Apache segfaulted at startup: statically linked OpenSSL 3 exported its symbols, and the box's httpd had libcrypto 1.0.0 loaded, so our calls bound to the old library. Legacy Lifeline turns that one-off rescue into a repeatable, agent-assisted pipeline.

What it does

1. Scans the box with read-only commands and fails on drift from its device profile.
2. Builds a PHP 8 Apache module against the box's own Apache, APR and glibc.
3. Runs seven ABI gates learned in the field: architecture, glibc ceiling, symbol isolation, RPATH, apr_off_t, runtime dependencies, CVE fixes present.
4. Runs old and new PHP side by side on a clone of the box (production port keeps serving) and diffs every response.
5. Packages one tarball per device with an idempotent POSIX installer that trials on a side port, backs up, switches, verifies and rolls back automatically.
6. Rehearses install, re-install and rollback on a staging clone.
7. Rolls out to the fleet only after a human approves the protected environment, and only to boxes whose SSH host key matches a human-recorded fingerprint.

A static scan lists every PHP construct the target version removed, so the app can be migrated in one round.

When any gate, test or rehearsal fails, the pipeline opens an incident issue with the report and the job link. A maintainer hands it to the Lifeline Incident Fixer, a GitLab Duo custom flow, with a one-line comment (supervised mode). The flow reads the report, the job log and our field rules, finds the root cause and opens a minimal fix MR. GitLab Duo Code Review then reviews every MR, and a human merges. If the agent's fix fails the pipeline again, the incident goes back to it (round 2, 3); after three rounds it stops and hands a human the full history.

We checked the repo against the real field upgrade and closed the gaps: the installer now runs Apache the way the vendor box does (httpd -f/-d with LD_LIBRARY_PATH), rewrites only the PHP lines of httpd.conf, renames the PHP 7 handler (otherwise PHP 8 serves .php files as plain text), and stops or rolls back if source is ever served. A second staging job rehearses on a clone laid out like the field box, an extension-parity gate compares php -m of the box with the build, and OpenSSL moved to the 3.5 LTS.

The same pipeline cross-builds the module for the real ARMv7 box (armel softfp, glibc ≤ 2.22) and runs the same seven gates on it; the CI-built package was then installed, re-run, rolled back and re-installed on a real box.

How we built it

• GitLab CI/CD with a device-clone image (Apache 2.4 prefork + mod_php 7.3), SAST and Secret Detection templates, generic package registry, environments, protected environment approvals, releases.
• GitLab Duo Agent Platform: a custom flow (ambient, Claude Sonnet) with GitLab API tools, plus .gitlab/duo/chat-rules.md that encodes the field knowledge: diagnosis order and what must never be loosened.
• Autonomy level: Supervised. Agents diagnose, fix and prepare releases. Shipping to customer devices stays a human decision.

Challenges

Making the agent useful instead of merely confident. The easy "fix" for a failing gate is to loosen the limit, edit the expected output, or skip host-key checks. We encoded the field lessons as explicit rules (never loosen a device limit; fix the app so it runs on both 7.3 and 8.x; never skip host-key verification) so the agent goes after the root cause instead. Another challenge: the device clone runs an archived Debian release, so the pipeline has to pull from archive.debian.org.

Accomplishments

Every gate maps to a real failure from the field rescue. The installer follows the same sequence used on the physical ARM box: precheck, trial on a side port, backup, switch, verify, rollback.

What we learned

The hard part of legacy upgrades is not writing code. It is proving the new binary is safe on a box you cannot rebuild. That proof is exactly what a pipeline plus an agent can carry.

What's next

Scan → clone → patch: generate the device clone itself (sysroot, matching Apache and APR) from the scan, so any customer box becomes a reproducible test target.

Built With

• apache
• bash
• gitlab
• gitlab-ci
• gitlab-duo
• gitlab-duo-agent-platform
• openssl
• php