• 

• 

  System Architecture: Integrating Dagster orchestration, Amazon S3, AWS RDS, Strands Decider 2B semantic routing, & Amazon Bedrock reporting

• 

  Bedrock-generated postmortem: structured summary outlining root cause, evidence findings, and recommended remediation actions.

• 

  Auditable AI decision log: Strands Decider 2B selects inspect_schema with recorded confidence and explainable reasoning.

• 

  Human-in-the-loop review: engineers inspect Bedrock postmortem summaries, add audit notes, and mark incidents resolved in UI.

• 

  Granular evidence inspection: automated triage identifies and flags the missing required currency column from upstream data.

• 

  Streamlit incident queue: newly logged contract breaches sit in pending status awaiting automated diagnosis or triage.

• 

  Validation contract failure in Dagster: raw Bronze data is preserved, Silver is blocked, and an incident ID is logged to RDS.

• 

  Healthy Dagster run: deterministic data validation passes, promoting raw Bronze orders directly to the Silver tier.

Agentic AWS DataOps: Project Ullo

Inspiration

Data pipelines can fail because of schema changes, missing values, and unexpected data quality issues. When this happens, engineers often need to inspect logs, identify the cause, decide what to do with affected data, and determine whether a pipeline can safely be retried.

I wanted to explore how agentic AI could help with incident investigation and decision-making without giving a model unrestricted control over pipeline operations. That idea became Project Ullo, an AI-assisted DataOps platform for investigating pipeline incidents and selecting appropriate actions through a defined set of tools.

What It Does

Project Ullo brings together data validation, pipeline orchestration, AI-driven tool selection, and incident reporting in one workflow.

Instead of relying on an AI model to execute arbitrary actions, the platform gives its decision engine a defined set of tools to choose from based on the available incident context.

The goal is to help data engineers investigate failures more efficiently while keeping decisions structured and observable.

How We Built It

The platform combines AWS services and open-source data engineering tools.

1. Data Storage: Amazon S3 organizes data into Bronze, Silver, and quarantine areas.

2. Data Validation: Python and pandas identify data quality issues, including excessive null values. The current validation rule uses a 15% null threshold per column.

3. Pipeline Orchestration: Dagster coordinates pipeline operations and helps manage the workflow.

4. AI Decision-Making: Strands Decider 2B evaluates the available context and selects an appropriate tool for the incident.

5. Controlled Tools: The decision engine can select from five tools:

• inspect_schema to inspect dataset structure.
• profile_nulls to analyze missing values.
• quarantine_batch to quarantine an affected batch.
• retry_pipeline to prepare a pipeline rerun configuration.
• alert_engineer to escalate an incident for human attention.

Invalid decisions fall back to alert_engineer with zero confidence instead of being treated as valid actions.

6. Incident Reporting: Amazon Bedrock supports report generation, while Streamlit provides a dashboard for reviewing workflow results. PostgreSQL/RDS supports the database layer.

Testing and Results

I evaluated the prototype through automated tests and a synthetic incident campaign.

• 24 automated tests passed in the documented test run.
• 15 synthetic test cases covered five valid scenarios and ten broken-data scenarios.
• All 15 cases matched their expected outcomes in the documented campaign.
• Clean and invalid datasets containing 1,100 rows were tested in the hosted environment.
• In one hosted test, Strands Decider selected inspect_schema with a confidence score of 0.927.

These results demonstrate the behavior observed in the tested scenarios. They do not establish production-scale reliability or guarantee successful recovery from every incident.

Challenges Faced

One of the main challenges was separating AI decision-making from actual pipeline operations. Selecting an appropriate tool is only part of the process; the system also needs clear tool definitions, validation rules, and safe fallback behavior.

Inference latency was another challenge. In a documented hosted test, Strands Decider took approximately 7 minutes and 3 seconds to select a tool on the configured CPU instance. This highlighted the importance of evaluating decision quality alongside inference performance.

The recovery workflow also has limitations. The retry_pipeline tool currently prepares a rerun configuration but does not automatically submit a new Dagster run. Quarantine-related IAM permissions also require further verification before production use.

What I Learned

Building Project Ullo showed me that an agentic system needs more than a model connected to tools. Reliable validation, explicit tool contracts, controlled permissions, useful reporting, and tests against expected outcomes are all essential.

I also learned to distinguish between a model selecting the expected action in a test and a system successfully recovering from an incident end to end. That distinction is especially important when building AI systems that interact with data infrastructure.

What's Next

The next steps are to improve inference latency, verify and tighten IAM permissions, expand incident test coverage, and implement a controlled end-to-end retry workflow with appropriate safeguards and human oversight.

The long-term goal is to help data engineers investigate pipeline incidents faster while keeping recovery actions observable and controlled.

Built With

• agentic
• ai
• amazon-bedrock
• amazon-web-services
• artificial-intelligence
• dagster
• data-engineering
• machine-learning
• pandas
• postgresql
• python
• strands-decider
• streamlit