AIOPS MLOps with Kubeflow Pipelines with Amazon SageMaker

AIOPS — MLOps Platforms with Kubeflow Pipelines with Amazon SageMaker AI Tool by Mahesh Marayya Chittibonu

Comment

Inspiration

The inspiration behind building the AIOps/MLOps hybrid pipeline with Kubeflow and Amazon SageMaker AI stems from addressing several fundamental challenges in enterprise machine learning engineering:

  1. Bridging Open-Source Agility with Managed Enterprise Scale The Problem: Data science teams love the open-source flexibility, customizability, and workflow orchestration of Kubeflow Pipelines (KFP) on Kubernetes, but managing raw compute clusters for heavy distributed training and high-concurrency model endpoints introduces immense operational overhead.

The Inspiration: Combine the best of both worlds—use Kubeflow as the open, developer-friendly orchestration plane on Amazon EKS, while offloading heavy-lifting tasks (such as distributed GPU training, hyperparameter tuning, and zero-downtime endpoint deployment) to Amazon SageMaker AI.

  1. Eliminating Infrastructure Friction & MLOps Silos The Problem: Machine learning workflows are frequently fragmented across disparate environments—data scientists work in isolated Jupyter Notebooks, ML engineers manually trigger training runs, and DevOps teams scramble to provision production inference infra.

The Inspiration: Standardize the end-to-end ML lifecycle into a single, automated, single-click deployment using Terraform, declarative IAM policies, and cloud-native Kubernetes gateway access.

  1. Guaranteeing Production Reproducibility & Governance The Problem: Moving models from experimentation to production often fails due to environment drift, unvetted IAM access, unencrypted storage, or missing pipeline audit trails.

The Inspiration: Enforce enterprise-grade security and governance out of the box using AWS IAM roles for service accounts (IRSA), encrypted Amazon S3 artifact stores, strict role segregation (data science vs. MLOps personas), and repeatable pipeline compilation (dsl-compile / KFP DSL).

  1. Hybrid Execution Efficiency & Cost Optimization The Problem: Running continuous heavy ML jobs on self-hosted Kubernetes nodes leads to idle GPU waste or out-of-memory cluster crashes during burst workloads.

The Inspiration: Orchestrate light tasks (data preprocessing, pipeline metadata tracking, initial exploration) directly on EKS/Kubeflow nodes, while dynamically spinning up fully-managed, pay-per-use SageMaker Training Jobs and Endpoints only when high-compute phases execute.

What it does

The AIOps / MLOps Hybrid Platform provides end-to-end automation for machine learning lifecycles by orchestrating open-source Kubeflow Pipelines (KFP) on Amazon EKS alongside fully managed Amazon SageMaker AI compute services. Key features and functional capabilities of the system include:1. One-Click Infrastructure ProvisioningInfrastructure-as-Code (IaC): Leverages Terraform to provision and configure an Amazon EKS cluster, worker node groups, VPC networking, security groups, and S3 artifact buckets in a single deployment step.Kubeflow on AWS Setup: Deploys a customized Kubeflow v1.6.1 manifest stack onto Amazon EKS with pre-configured ingress gateways, multi-user namespace isolation, and local storage mappings.2. Hybrid Execution & Offloaded ComputeKubeflow as the Orchestration Engine: Acts as the central pipeline controller where developers author, schedule, and monitor ML experiments via the Kubeflow UI dashboard or Python KFP SDK. SageMaker Execution Components: Moves compute-heavy tasks away from the Kubernetes worker nodes into managed SageMaker infrastructure:Data Preprocessing & Feature Engineering: Executes distributed data prep jobs via SageMaker Processing. Model Training & Tuning: Triggers Amazon SageMaker Training and Hyperparameter Optimization (HPO) jobs using spot or dedicated GPU instances. Model Deployment: Provisions auto-scaling, highly available SageMaker Real-Time Inference Endpoints directly from pipeline steps.3. Enterprise Access Control & GovernanceAWS IAM & Fine-Grained RBAC: Implements strict identity policies and IAM Roles for Service Accounts (IRSA).Role Segregation: Enforces secure boundaries between Kubernetes pod identities, gateway instances, SageMaker execution roles, and Amazon S3 bucket storage.KMS Encryption: Secures underlying Amazon EBS volumes, EKS secrets, and S3 pipeline artifacts using AWS Key Management Service (KMS).4. Interactive Data Science & Distributed TrainingManaged Jupyter Notebooks: Provides containerized JupyterLab environments on EKS with pre-installed deep learning frameworks (PyTorch, TensorFlow) and pre-configured SageMaker Python SDKs. Distributed PyTorch Training (DDP): Supports PyTorch Distributed Data Parallel (DDP) scripts for multi-node, multi-GPU model training jobs.5. Secure Gateway & Dashboard AccessEC2 Kubeflow Gateway: Uses an EC2 gateway instance with SSH tunneling and port-forwarding to restrict access to the Kubeflow dashboard, preventing public exposure of internal pipeline management UIs.Pipeline Compilation & Versioning: Allows data scientists to write Python pipeline definitions (*.py), compile them into tarball components via dsl-compile, and version runs across dedicated user namespaces.

How we built it

Here is how the AIOps/MLOps Kubeflow and Amazon SageMaker Hybrid Platform was built:

  1. Infrastructure & Core Orchestration Setup (IaC & EKS) Terraform & Automation: Built modular Terraform configurations (deployments/vanilla/terraform) to automatically provision the foundational AWS infrastructure—including Virtual Private Clouds (VPCs), subnets, security groups, KMS encryption keys, and Amazon EKS worker node groups.

Kubernetes Gateway Instance: Provisioned an Ubuntu 20.04 LTS EC2 gateway instance (kubeflow-gateway) attached with a dedicated IAM role (kubeflow-gateway-role) to act as the primary operational control plane for CLI commands, UI management, and remote code deployment.

Kubeflow Manifests & Deployment: Deployed Kubeflow v1.6.1 on Amazon EKS using customized AWS manifest releases (v1.6.1-aws-b1.0.1), utilizing make deploy to orchestrate ingress gateways, multi-user namespace isolation, and service meshes.

  1. Identity, Access Management & Security (IAM & IRSA) Role-Based Least Privilege: Configured granular AWS IAM policies (kubeflow-gateway-eks, kubeflow-kms-admin) alongside managed service policies (CloudWatch, IAM, S3, EC2 full access) to ensure proper API communication.

Pod-Level Permissions: Automated service account authorization using utility scripts (kubeflow_iam_permissions.sh and sagemaker_role.sh) to bind Kubernetes namespaces directly to AWS IAM roles. This allows Kubeflow pipeline components and Jupyter notebooks to securely make AWS API calls to SageMaker and S3 without exposing static access keys.

  1. Pipeline Authoring & Compilation Engine Kubeflow Pipelines SDK (kfp): Utilized the KFP Python SDK and domain-specific language (dsl-compile) to author end-to-end machine learning workflows (such as the MNIST Kmeans classification pipeline).

Component Modularization: Structured the workflow into distinct, containerized execution blocks:

Data Preprocessing Component: Runs feature engineering tasks (e.g., kmeans_preprocessing.py) and stages processed datasets into Amazon S3 buckets (dl-demo-kubeflow-...).

SageMaker Training Step: Programmatically dispatches managed Amazon SageMaker Training Jobs leveraging custom Docker containers and dedicated training datasets.

Model Deployment Component: Automates the creation of persistent Amazon SageMaker Real-Time Inference Endpoints.

  1. Interactive Data Science & Distributed Training Execution Jupyter Notebook Workspaces: Deployed containerized Jupyter-PyTorch DLC images (jupyter-pytorch:1.12.0-cpu-py38-ubuntu20.04) directly on EKS for data scientists to experiment, test code, and clone repositories (aws-samples/eks-kubeflow-cloudformation-quick-start).

Distributed Data Parallel (DDP): Integrated PyTorch DDP training scripts (cifar10-distributed-gpu-final.py) to distribute deep learning workloads seamlessly across multi-node CPU/GPU worker pools.

  1. Secure Dashboard Access & CI/CD Tooling Secure Port-Forwarding Tunnel: Established secure access to the Kubeflow UI dashboard (localhost:9005) by combining Kubecontext configuration, make port-forward, and an encrypted SSH tunnel (-L parameter) originating from the gateway EC2 instance.

Version-Controlled Repository: Maintained modular MLOps scripts, infrastructure scripts, and pipeline definitions under a centralized Git repository structure (github.com/mchittibonu/mlops).

Challenges we ran into

Building and deploying a hybrid MLOps platform combining open-source Kubeflow on Amazon EKS with managed Amazon SageMaker AI presented several complex technical hurdles. Here are the primary challenges we encountered and how we resolved them:

  1. Kubernetes & AWS IAM Permission Alignment (IRSA Complexity) The Challenge: Kubeflow runs native Kubernetes pods inside Amazon EKS, but those pods need to securely make AWS API calls to trigger SageMaker training jobs and read/write datasets to Amazon S3. Managing static AWS credentials inside Kubernetes secrets introduced massive security risks and compliance gaps.

How We Solved It: We implemented AWS IAM Roles for Service Accounts (IRSA). By developing automated permission-binding scripts (kubeflow_iam_permissions.sh), we mapped Kubernetes service accounts in specific namespaces (e.g., kubeflow-user-example-com) directly to dedicated AWS IAM roles, ensuring least-privilege access without hardcoded credentials.

  1. Network Isolation & Secure Kubeflow Dashboard Access The Challenge: Exposing the Kubeflow multi-user dashboard and Jupyter notebooks securely without exposing sensitive internal pipeline management UIs to the public internet or running into complex enterprise firewall and VPN bottlenecks.

How We Solved It: We bypassed traditional public load balancer exposure by establishing an isolated EC2 Gateway instance (kubeflow-gateway). We utilized secure SSH tunneling with local port forwarding (-L 9005:localhost:8080) combined with make port-forward commands, ensuring that all traffic to the Kubeflow UI remained fully encrypted and strictly controlled.

  1. Orchestrating Hybrid Compute Boundaries (EKS vs. SageMaker) The Challenge: Determining the exact boundary between what executes on the local EKS Kubernetes worker nodes versus what gets offloaded to managed SageMaker infrastructure. Misconfigured compute boundaries risked either overloading the EKS cluster memory during heavy model training or adding unnecessary network latency for lightweight preprocessing tasks.

How We Solved It: We architected pipelines to use EKS strictly for lightweight tasks—such as workflow orchestration, pipeline compilation (dsl-compile), and interactive Jupyter notebook exploration—while offloading all resource-intensive data prep, distributed training, and model hosting entirely to Amazon SageMaker managed instances.

  1. Dependency Management & Pipeline Compilation Friction The Challenge: Version mismatches between the Kubeflow Pipelines SDK (kfp), local Python environments (python3-pip), and the underlying AWS SageMaker operator libraries frequently caused pipeline tarball compilation failures (dsl-compile) or container registry authentication errors.

How We Solved It: We standardized our deployment stack by locking down specific release versions (Kubeflow v1.6.1 and AWS-optimized manifests v1.6.1-aws-b1.0.1) and containerizing the development workspace using pre-built Jupyter-PyTorch Deep Learning Containers (DLCs) to guarantee identical runtime dependencies across environments.

Accomplishments that we're proud of

Here are the key accomplishments we are most proud of in building the AIOps/MLOps Hybrid Platform (Kubeflow on Amazon EKS + Amazon SageMaker AI):

  1. Seamless Hybrid Architecture Integration Bridging Open-Source & Managed Cloud: Successfully unified the flexibility and customizability of open-source Kubeflow Pipelines on Amazon EKS with the heavy-lifting, enterprise-grade scale of Amazon SageMaker AI training and inference endpoints.

Optimized Compute Boundaries: Successfully offloaded resource-intensive tasks (like distributed GPU training and hyperparameter optimization) to SageMaker while keeping workflow orchestration and interactive notebooks lightweight on EKS.

  1. Enterprise-Grade Security & Least-Privilege Governance Secure IRSA Implementation: Eliminated static, high-risk credentials by successfully integrating AWS IAM Roles for Service Accounts (IRSA), mapping Kubernetes pod identities securely to AWS API permissions for S3, EKS, and SageMaker.

Encrypted Data & Compute Foundations: Secured sensitive pipeline artifacts, container volumes, and cluster states using AWS KMS encryption and granular AWS managed policies.

  1. Production-Ready Infrastructure-as-Code (IaC) & Automation One-Click Deployments: Engineered modular, repeatable Terraform configurations and automated Makefile routines that provision VPCs, EKS clusters, and supporting AWS infrastructure seamlessly.

Streamlined CI/CD Pipeline Compilation: Successfully automated pipeline building, versioning, and tarball compilation using the Kubeflow Pipelines SDK (dsl-compile), reducing manual deployment friction.

  1. Secure Operational Access Control Zero-Trust UI Exposure: Designed a secure, non-public operational workflow utilizing an isolated EC2 Gateway instance combined with encrypted SSH tunneling and local port-forwarding (-L 9005:localhost:8080) to safely access the Kubeflow dashboard without exposing internal components to the public internet.

What we learned

Here are the key technical and architectural insights we learned while building and deploying the AIOps/MLOps Hybrid Platform (Kubeflow on Amazon EKS + Amazon SageMaker AI):

  1. The Power of Hybrid Compute Separation What We Learned: Trying to run heavy, compute-intensive model training and batch processing directly on standard Kubernetes worker nodes quickly leads to resource contention, out-of-memory (OOM) cluster crashes, and high infrastructure costs.

The Takeaway: Separating concerns—using Kubeflow on EKS strictly as the agile workflow orchestrator and interactive notebook plane, while offloading heavy data prep, distributed GPU training, and real-time endpoint hosting entirely to Amazon SageMaker—provides the ultimate balance of developer flexibility and production stability.

  1. Modernizing Security with IRSA Over Static Credentials What We Learned: Relying on hardcoded AWS access keys or shared cluster-wide credentials for Kubernetes pods creates massive security vulnerabilities and audit nightmares in enterprise environments.

The Takeaway: Implementing AWS IAM Roles for Service Accounts (IRSA) is essential for zero-trust cloud-native architectures. Binding specific Kubernetes namespaces and service accounts directly to least-privilege IAM roles ensures that pipeline pods securely authenticate with S3, EKS, and SageMaker without exposing sensitive secrets.

  1. Simplifying Complex Deployments with Repeatable Automation What We Learned: Deploying multi-component open-source platforms like Kubeflow manually across cloud environments introduces severe configuration drift and environment incompatibilities.

The Takeaway: Combining declarative Terraform scripts, locked software release versions (Kubeflow v1.6.1 / AWS manifests), and automated Makefile routines transforms a complex, multi-day manual installation into a reliable, repeatable, one-click deployment blueprint.

  1. Securing Operational Access Without Public Exposure What We Learned: Exposing internal MLOps dashboards and management UIs via standard public load balancers opens unnecessary attack surfaces and violates compliance standards.

The Takeaway: Utilizing an isolated EC2 Gateway instance paired with encrypted SSH local port-forwarding creates a secure, zero-trust operational tunnel. This allows data scientists and engineers to safely access the Kubeflow UI and Jupyter workspaces without exposing internal cluster endpoints to the public internet.

What's next for AIOPS MLOps with Kubeflow Pipelines with Amazon SageMaker

Looking ahead, the evolution of our AIOps/MLOps Hybrid Platform (Kubeflow on Amazon EKS + Amazon SageMaker AI) focuses on scaling automation, tightening enterprise governance, and integrating modern Agentic AI workflows.

Here is what is next for the platform:

  1. Native Agentic AI & LLM Fine-Tuning Integration Next Step: Embed large language model (LLM) fine-tuning and retrieval-augmented generation (RAG) pipelines directly into Kubeflow workflows, leveraging high-performance inference tools (such as NVIDIA NIM and Nebius Token Factory) alongside SageMaker training jobs.

Why: Moving beyond traditional tabular/vision machine learning models (like MNIST/CIFAR-10) to support autonomous agents, LLM guardrails, and automated evaluation loops within the same MLOps pipeline.

  1. GitOps-Driven MLOps (ArgoCD & Continuous Delivery) Next Step: Transition from manual make deploy pipeline uploads and Terraform scripts to a fully automated GitOps workflow using ArgoCD for continuous deployment of Kubeflow manifests and infrastructure changes.

Why: Ensuring that any update to pipeline definitions, model code, or cluster configurations is automatically validated, tested, and promoted across Dev, QA, UAT, and Prod environments.

  1. Automated Drift Detection & Continuous Retraining (AutoMLOps) Next Step: Integrate automated monitoring tools (such as Amazon SageMaker Model Monitor and Prometheus/Grafana on EKS) to track data drift, concept drift, and prediction latency in production endpoints.

Why: Automatically trigger Kubeflow retraining pipelines and execute shadow deployments when performance degrades, closing the loop on continuous autonomous operations (AIOps).

  1. Advanced FinOps & Cost Optimization Next Step: Implement dynamic resource scheduling leveraging Kubernetes Karpenter and SageMaker Spot Instances to automatically scale worker nodes down to zero during idle hours and optimize GPU allocation based on workload demands.

Why: Maximizing cloud cost-efficiency without sacrificing performance during peak training or batch inference bursts.

Built With

Share this project:

Updates

Submission history