Schema Sentinel-Strands

• 

  Real-time SRE observability dashboard tracking conformed records, auto-healed drifts, 100% SLA, and AWS Bedrock Mantle inference latency.

• 

  Interactive DAG visualizer tracking B01-B05 stream partitions with live execution telemetry intercepting real-time schema drift events.

• 

  Side-by-side JSON transformation audit showing raw drifted payloads conformed to warehouse schema with AST-verified Python code patches.

• 

  Interactive test playground allowing judges to inject custom malformed JSON and watch Strands Agent synthesize real-time code fixes.

💡 Inspiration: The 2:00 AM Problem

Every data engineer knows this nightmare:

1. At 2:00 AM, an external API silently renames a single JSON key (e.g., author becomes owner_name).
2. The ingestion pipeline violently crashes. PagerDuty alarms go off.
3. Downstream dashboards and ML models freeze. An engineer wakes up in the middle of the night just to write a 2-line regex fix.

Why should production pipelines be so fragile?

We built SchemaSentinel-Strands to turn pipelines from brittle glass into self-healing infrastructure: a 24/7 AI Reliability Engineer that intercepts breaking schema drift and repairs data in-flight without dropping a single record.

🛡️ What It Does: 5-Stage Autonomous Healing

When breaking schema drift hits the warehouse, SchemaSentinel heals it in <400ms through 5 visual steps:

1. 🚨 Intercept: A malformed record fails the strict SQLite schema contract. Instead of crashing, the pipeline catches the error and isolates the payload.
2. 🧠 Reason: The Strands Agent inspects the failing JSON, compares it with the warehouse schema, and understands column intent using semantic scoring.
3. ⚡ Synthesize: Powered by AWS Bedrock Mantle (xai.grok-4.6), the agent writes a pure Python transformation patch (transform_record()) in real time.
4. 🔒 Secure AST Sandbox: The code is compiled in an isolated Python Abstract Syntax Tree (AST) sandbox. Any unsafe call (os, sys, eval) is blocked instantly.
5. ✨ Conformed Ingress: Numbers, nested arrays, and aliases are normalized to a 0–100 scale and committed to SQLite with a 100% Reliability SLA.

⚙️ How I Built It

• Autonomous Agent Brain: Strands Agents SDK invoking AWS Bedrock Mantle (xai.grok-4.6) for sub-second code generation.
• AST Security Barrier: Python ast syntax parser enforcing strict memory and namespace isolation.
• Chaos Ingress Engine: Real-time multi-partition stream generator simulating real-world upstream API drift.
• SRE Command Center: A modern Streamlit dashboard featuring live DAG stage nodes, real-time telemetry logs, JSON Before/After diffs, and an interactive Judge Playground.

🧗 Challenges & Breakthroughs

• No Hardcoded Rules: We eliminated static string checks. The agent uses recursive leaf scoring to differentiate a repo star count (47k) from an actual project rating (4.6/5).
• Zero-Trust Security: Executing LLM-generated code live in production is dangerous. Our AST sandbox inspects code structure before compilation, with an instant deterministic fallback if compilation fails.
• Sub-Second Speed: Optimized prompts and Bedrock Mantle inference achieve full drift-detection to warehouse-commit in ~380ms.

🧠 Key Takeaway

Agents shouldn't just be conversational chatbots. In SchemaSentinel, the agent acts as active, self-healing system middleware—writing and executing micro-patches at runtime to eliminate infrastructure downtime.

🚀 What's Next

• Native connectors for Snowflake, BigQuery, and Databricks.
• Automated GitHub Pull Request generation to update upstream dbt models permanently.
• Production Kafka / Apache Flink streaming consumer integration.

Built With

• ast
• autonomous-agents
• aws-bedrock
• aws-bedrock-mantle
• data-engineering
• etl
• grok-4.6
• pandas
• python
• react
• rest-api
• sqlite
• strands-agents-sdk
• streamlit
• tailwind-css
• tavily-api
• typescript