Devpost

AI Voice Agent on AWS EKS

Inspiration

We were inspired to build a real-time AI Voice Agent that could automate voice-based interactions and improve customer support experiences. Our goal was to create a scalable, reliable system that could process voice input, understand user requests using AI, and respond naturally with generated speech. We wanted to explore how cloud-native technologies could support real-time communication and handle increasing numbers of concurrent voice sessions.

What it does

The AI Voice Agent enables real-time, bidirectional communication between users and AI services through WebSockets. It integrates Speech-to-Text (STT), Large Language Models (LLMs), and Text-to-Speech (TTS) to convert voice input into meaningful spoken responses.

The application runs on AWS EKS and uses Kubernetes to manage workloads and application availability. AWS load balancing and ingress components handle incoming traffic, while Redis supports application data-handling requirements. Prometheus and Grafana provide monitoring and visibility into infrastructure and application metrics.

How we built it

We deployed the application on Amazon EKS using Kubernetes Deployments and Services. We configured an Application Load Balancer and Ingress to route external traffic and support secure WebSocket communication.

We integrated STT, AI processing through services such as OpenAI and Gemini, and TTS to enable end-to-end voice interactions. We used Redis for supporting application state and data handling.

For observability, we worked with Prometheus and Grafana to monitor application and infrastructure health. We also explored custom metrics, including active WebSocket connections, to understand real application demand.

We evaluated Kubernetes HPA for scaling gateway pods, KEDA for suitable event-driven worker workloads, and node autoscaling approaches to support changing workloads.

Challenges we ran into

WebSocket connectivity: Configuring reliable, secure WebSocket communication through the load balancer and ingress layer required careful troubleshooting of routing, TLS certificates, and connection timeouts.
Concurrent sessions: CPU and memory usage alone did not accurately represent the workload generated by persistent WebSocket connections.
Autoscaling: We explored how application-specific metrics could help scale workloads according to actual demand rather than relying exclusively on resource utilization.
Monitoring: We needed better visibility into application behavior and infrastructure health to identify performance bottlenecks.
End-to-end latency: Voice processing depends on several stages—STT, AI inference, and TTS—so identifying delays required considering the entire request pipeline.

Accomplishments that we're proud of

Deployed an AI Voice Agent architecture on AWS EKS.
Integrated real-time WebSocket communication with AI-powered voice-processing components.
Worked on load balancing and secure communication for persistent connections.
Established monitoring using Prometheus and Grafana.
Explored custom application metrics to support more meaningful scaling decisions.
Developed a stronger understanding of Kubernetes autoscaling, cloud networking, and the operational requirements of real-time AI applications.

What we learned

We learned that building a reliable AI Voice Agent requires more than deploying an application. It involves understanding network connectivity, persistent connections, latency, observability, and workload behavior.

We gained practical knowledge of AWS EKS, Kubernetes, WebSockets, load balancing, Prometheus, Grafana, and autoscaling strategies. We also learned that application-specific metrics, such as active connections and request latency, can provide better insight into scaling requirements than CPU and memory metrics alone.

This project strengthened our ability to troubleshoot distributed systems and design infrastructure for scalable, observable, and reliable real-time AI applications.