AETHER-OMNI: Self-Healing Swarm Engine

Autonomous context-aware self-healing swarm engine for schema drift detection, process isolation, sandbox validation, and cryptographic provenance.

Comment

💡 Inspiration

In enterprise data environments, silent schema drift can break downstream pipelines when an upstream field changes unexpectedly. For example, renaming user_id to account_uuid can cause SQL queries, ML feature pipelines, and dashboards that depend on the original schema to fail.

We built AETHER-OMNI to explore a faster response workflow: detect schema drift, stop or isolate the affected execution path, analyze the broken dependency, generate a repair, validate it in a sandbox, and produce an auditable record of the recovery.

The project uses metadata-style context, schema inspection, lineage impact modeling, autonomous repair logic, sandbox validation, and cryptographic provenance to demonstrate an end-to-end self-healing workflow.

🚀 What It Does

AETHER-OMNI is a context-aware self-healing pipeline prototype that detects a schema change and orchestrates a recovery workflow.

Key Capabilities

  • Schema Context & Inspection: Builds a dataset URN and attempts to query DataHub through GraphQL, with a native SQLite schema inspection fallback for local execution.

  • Schema Drift Detection: Detects differences between the expected schema contract and the live SQLite schema, including breaking field renames such as user_id → account_uuid.

  • Process Circuit Breaker: Uses the pipeline PID to stop or freeze the affected process using OS-level process controls. In our submitted benchmark run, the circuit breaker completed in approximately 2.41ms.

  • Impact Analysis: Identifies the broken SQL reference and produces an impact report showing modeled downstream ML and dashboard dependencies.

  • Autonomous Healing: Generates a repaired version of the pipeline by replacing the broken schema reference with the detected replacement field.

  • Sandbox Validation: Creates an isolated SQLite sandbox containing 1,000 synthetic records and validates that the repaired query executes successfully.

  • Cryptographic Provenance: Generates a SHA-256 hash and nanosecond-resolution timestamp for the healed artifact.

  • Recovery Artifacts: Produces a pull-request-style repair manifest and a DataHub mutation record that can be inspected as part of the recovery workflow.

🛠️ How We Built It

AETHER-OMNI is built as a multi-stage recovery pipeline connecting metadata context with native process execution.

1. Context & Metadata Layer

We created a DataHubMCPClient that:

  • Constructs standardized DataHub dataset URNs.
  • Attempts to retrieve dataset schema information through DataHub's GraphQL endpoint.
  • Falls back to direct SQLite schema inspection when a local DataHub service is unavailable.
  • Exports the collected context into a JSON manifest for downstream agents.

2. Sentinel Detection Agent

The Sentinel agent compares the expected schema contract with the live database schema.

When drift is detected, it:

  • Identifies missing and newly introduced fields.
  • Locates the SQL line referencing the broken field.
  • Generates an impact-analysis artifact.
  • Models the downstream blast radius across ML models and dashboards.

3. Native Process Circuit Breaker

The circuit breaker reads the active pipeline PID and uses psutil together with platform-specific process controls to halt the affected execution path.

This allows the demo workflow to isolate the pipeline before continuing with automated repair.

4. Surgeon Healing Agent

The Surgeon agent reads the impact-analysis output and generates a repaired pipeline.

For the demonstrated schema drift:

user_id → account_uuid
the agent rewrites the affected SQL query to:

SELECT account_uuid AS user_id, account_balance
FROM user_transactions
LIMIT 1;
5. Native Sandbox Validation

The repaired workflow is tested against an isolated SQLite database containing 1,000 synthetic records.

The validation confirms that the repaired query can execute successfully against the updated schema without modifying the original production database.

6. Provenance & Recovery Records

After validation, AETHER-OMNI:

Generates a SHA-256 provenance hash.
Records a nanosecond-resolution timestamp.
Writes a provenance manifest.
Generates a pull-request-style repair artifact.
Attempts a DataHub GraphQL mutation and records the resulting response or local fallback result.
7. Frontend Control Room

The frontend is built with Next.js, React, TypeScript, TailwindCSS, and Three.js.

It visualizes the recovery lifecycle:

ACTIVE → DRIFT → FROZEN → SANDBOX TESTING → HEALED

⚡ Challenges We Faced
Reliable Schema Drift Detection

A major challenge was distinguishing between a healthy schema and a breaking schema change while keeping the demonstration reproducible. We solved this by maintaining an expected schema contract and comparing it with the live SQLite table structure.

Safe Process Intervention

Stopping an active process safely is platform-dependent. We implemented a native process shield using psutil and platform-specific process controls to demonstrate how an execution path can be isolated during recovery.

Autonomous Code Repair

The healing agent needed to modify only the affected query while preserving the rest of the pipeline. We connected the Sentinel's detected field change to a targeted source-code transformation.

Reproducible Sandbox Testing

We wanted the repair to be validated independently of the production database. The solution was a temporary SQLite sandbox containing 1,000 synthetic records.

Working With Optional External Services

A local DataHub instance may not always be available during hackathon evaluation. To keep the project runnable, the metadata client attempts a DataHub GraphQL request and falls back to native SQLite inspection for the local demonstration.

🏆 Accomplishments That We're Proud Of
Built an end-to-end schema drift demonstration from detection through repair and validation.
Implemented OS-level process intervention using a PID-based circuit breaker.
Achieved approximately 2.41ms circuit-breaker execution latency in the submitted benchmark run.
Detected a breaking user_id → account_uuid schema change and located the affected SQL reference.
Generated an autonomous repaired pipeline.
Validated the repair using 1,000 synthetic SQLite records.
Generated SHA-256 provenance artifacts with nanosecond-resolution timestamps.
Built a real-time visual control room to represent the recovery lifecycle.
Created inspectable JSON artifacts for context retrieval, impact analysis, circuit breaking, repair provenance, and recovery records.
📚 What We Learned
Context is essential for autonomous agents. A repair system needs to understand the schema and the affected code before attempting a fix.
Metadata systems can provide useful context for reasoning about datasets and their dependencies.
Autonomous repair should include verification rather than assuming that generated code is correct.
A reproducible local fallback can make a hackathon project easier for judges to run and inspect.
Provenance artifacts are valuable for making automated changes more auditable.
🔮 What's Next for AETHER-OMNI
Connect the impact analysis to a live DataHub lineage graph instead of modeled downstream assets.
Replace the current PR-style artifact with live GitHub API pull request creation.
Extend the healing engine from targeted field replacement to broader AST-based code transformations.
Add policy and human-approval gates before production changes are applied.
Support additional databases and cloud data platforms.
Expand the agent workflow to handle more types of schema evolution and cross-platform migrations.
Run benchmark measurements across multiple environments to establish reproducible latency results.

Built With

Share this project:

Updates