CAF (Cloud AI-Agentic FrameWork)

An AI Operating System for developers—build production-ready AI agents with automated memory, knowledge, tools, and intelligent model orchestration behind a simple API.

Comment

CAF (Cloud AI-Agentic FrameWork)

Inspiration

The inspiration for CAF came from watching developers continually struggle with the overhead of integrating AI into their applications. Whenever a developer wants to build a production-grade AI agent, they are forced to deal with a mountain of complexity: managing vector databases, writing chunking algorithms for RAG, engineering prompts, orchestrating tools, handling context limits, and building session memory.

It felt like every team was reinventing the wheel. The gap between a simple LLM API call and a production-grade agent was just too vast. I realized that developers shouldn't have to be machine learning engineers or infrastructure experts just to add an intelligent chatbot to their app. We needed a true Cloud Runtime for AI Agents—a platform where all the complex infrastructure, graph-based memory, document parsing, and provider integrations are handled entirely in the cloud, allowing developers to focus purely on building their core applications.

What it does

CAF (formerly OrchestraAI) acts as a centralized AI runtime and intelligent execution engine for your applications. Instead of dealing with raw database connections or complex prompts, your application communicates with CAF via a single, unified API.

CAF provides a beautiful drag-and-drop web console where you can:

  1. Plug in your API keys for multi-provider support (Gemini, Groq, OpenAI, Anthropic, Ollama).
  2. Create strictly isolated AI Agents, where each agent securely owns its dedicated memory, API keys, and knowledge base.
  3. Upload Knowledge Documents (like .docx, .pdf, etc.) which are instantly parsed, chunked, and embedded.

When an end-user chats with the agent, CAF automatically handles autonomous web searches (via DuckDuckGo), retrieves local knowledge via Hybrid Search, maintains session memory, and orchestrates the best model for the job, all behind a simple REST API.

How we built it

  • Backend: Built with Python, FastAPI, and Uvicorn to handle asynchronous, high-performance API requests.
  • Knowledge Pipeline: We integrated FAISS for lightning-fast, local vector similarity search. When a document is uploaded, the pipeline automates text extraction, optimal chunking, and embedding generation.
  • Memory & Graph: Engineered a robust local SQLite architecture to handle session states, document metadata, and relationship mappings.
  • Frontend Console: A sleek, vanilla JavaScript and CSS dashboard featuring modern glassmorphism UI/UX principles to make agent and document management a breeze.
  • AI Integrations: Deeply integrated with models like Gemini 1.5 and Groq (Llama 3) for high-speed inference, along with dynamic fallback routing to handle rate limits perfectly.

Challenges we ran into

One of the biggest hurdles was managing context windows and token limits. When an end-user chats with an agent for a long time, feeding the entire conversation history back to the LLM becomes too expensive and eventually crashes the application. To solve this, we built a background summarization engine that constantly condenses older messages without losing critical context, saving the original chat for later retrieval.

Another massive challenge was infrastructure state synchronization. We had issues where deleting a document wouldn't properly cascade and delete the chunked vectors in FAISS, causing hallucinated answers from "ghost" documents. We had to completely engineer a robust cascading delete system that ensures local database records and FAISS indices stay perfectly synchronized.

Accomplishments that we're proud of

We are incredibly proud of the Developer Experience and Agent Isolation. By shifting all the heavy lifting to the CAF cloud runtime, a developer can literally integrate a fully autonomous, web-searching, RAG-enabled AI agent into their terminal, mobile app, or website using just standard HTTP request code.

Furthermore, we successfully implemented strict multi-tenant isolation. Each agent operates in its own sandbox with dedicated memory, vectors, and execution policies, meaning zero cross-contamination between different business logic use-cases.

What we learned

We learned that the true bottleneck in AI adoption isn't the intelligence of the models themselves, but the orchestration surrounding them. We realized that AI applications should not manage prompts, memory, retrieval, execution, or orchestration—just as modern applications do not manually manage CPU allocation or memory addressing inside an operating system. Applications should simply request intelligence, and the runtime should handle everything else.

What's next for CAF (Cloud AI-Agentic FrameWork)

  • Autonomous Planner & Reflection: Building a v3 engine where agents can reflect on their own mistakes and create multi-step execution plans before responding to the user.
  • Distributed Execution Engine: Allowing the platform to determine the safest environment for work to execute (e.g., executing code in a secure Cloud Terminal or headless browser).
  • Advanced Knowledge Graphs: Expanding the hybrid retrieval pipeline to construct complex entity-relationship graphs alongside standard vector indexes.
  • SDKs: Building official lightweight SDKs for Node.js, Python, and Go to make interacting with the CAF runtime even simpler for enterprise teams.

Built With

  • ai-infrastructure
  • artificial-intelligence
  • cloud-runtime
  • css3
  • faiss
  • fastapi
  • gemini-api
  • groq
  • html5
  • knowledge-graphs
  • large-language-models
  • multi-agent-systems
  • python
  • rag-(retrieval-augmented-generation)
  • rest-api
  • sqlite
  • uvicorn
  • vanilla-javascript
  • vector-databases
  • web-search-integration
Share this project:

Updates