Kalavai

The first devops agent to help you deploy production-grade AI models

Comment

Inspiration

Today, 50% of enterprises are deploying open source AI models. And this number keeps growing. In them, AI developers waste 65% of their time in non-AI engineering tasks such as boilerplate deployment code and infrastructure management. I should know, I was one of them. Leading a team of 5, I grew frustrated at the slow pace our many prototypes hit production.

Devops for AI is only getting more complex. AI devs need help.

What it does

Our platform, Kalavai, helps developers focus on the value-add activities of GenAI by handling compute provisioning, networking, deployment and exposing models via API, and does it at the scale.

However, our API-based functionality still requires deep expertise from the users. To make GenAI more approachable, we built Genie, the first Devops agent specifically trained to be the only deployment engineer you'll ever need in your projects.

Say “deploy Llama3 to Google Cloud and on-prem, optimized for latency,” and Genie handles it end-to-end—no YAML, no cloud certs.

How we built it

Built on our existing hybrid cloud Kalavai API and our OpenAI compatible open source inference platform CoGen AI, Now users can deploy in seconds via natural language whilst maintaining full control via our API and dashboards.

The tech stack behind Genie is simple:

  • Streamlit for the UI
  • Qwen2.5-7B-instruct on CoGen AI as the agent brains for tool calling
  • Our Kalavai API for backend and cloud operations.

Challenges we ran into

Taming agents is not an easy feat! Hallucinations and non-deterministic behaviour are key challenges to overcome, particularly in environments where we want to give the users a consistent experience.

Accomplishments that we're proud of

Being able to easily plug the new agent to our existing Kalavai API has been a highlight. What before took config files and API calls now can be done with a simple English command.

Dogfooding our own platform CoGen AI has also been a delight!

What's next for Kalavai

This is the beginning of both Kalavai and Genie. We have plenty on the works!

  • Extend Genie's support for all our Kalavai API functionality, including creation of on prem clouds
  • Make Genie more robust and able to flag issues
  • Expand Kalavai to support agent deployment

Stay tuned!

Built With

Share this project:

Updates