Nvidia-AWS-Deep-Agent

- "Transparent AI reasoning for intelligent document research" - "RAG with a brain—multi-step reasoning you can trust" - "Your research assistant that thinks out loud"

Comment

NIM RAG Agent: Building an Intelligent Research Assistant

💡 Inspiration

The inspiration for this project came from a simple observation: traditional RAG systems are black boxes. When you ask a question, they search, retrieve, and respond—but you never see how they arrived at their answer. As developers and researchers, we needed something more transparent, more intelligent, and more trustworthy.

I wanted to build a system that doesn't just answer questions—it thinks out loud. A system that can:

  • Break down complex queries into manageable steps
  • Show its reasoning process in real-time
  • Use multiple tools strategically (not just document retrieval)
  • Cite its sources with confidence scores
  • Learn and adapt as new documents are added

The NVIDIA NIM + AWS Hackathon provided the perfect opportunity to combine NVIDIA's powerful language models with AWS's scalable infrastructure to create a production-ready intelligent research assistant.

🎯 What It Does

The NIM RAG Agent is an agentic RAG (Retrieval-Augmented Generation) system that goes beyond simple question-answering:

Core Capabilities

  1. Multi-Step Reasoning: Uses LangGraph to plan and execute complex queries that require multiple reasoning steps
  2. Transparent Planning: Shows users the agent's thought process, including which tools it plans to use and why
  3. Tool Integration: Combines document retrieval with computational tools (like a calculator) for versatile problem-solving
  4. Real-Time Feedback: Provides progress updates as documents are processed and embedded
  5. Source Attribution: Every answer includes citations with relevance scores, building trust through transparency
  6. Document Management: Interactive document viewer showing all 683+ indexed documents with search capabilities

Unique Features

  • API Call Logger: Track every NVIDIA NIM API call with timestamps, costs, and response details
  • Enhanced UI: Modern Streamlit interface with tabbed navigation, real-time updates, and visual feedback
  • Flexible Document Processing: Supports PDF, DOCX, TXT, and Markdown files with automatic chunking
  • Production-Ready: Designed for AWS EKS deployment with autoscaling, monitoring, and health checks

🏗️ How I Built It

Architecture Overview

The project follows a modular, production-ready architecture:

User Interface (Streamlit)
         ↓
   RAG Agent (LangGraph)
    ↙         ↘
NIM LLM    Vector Store (ChromaDB)
              ↓
      NIM Embeddings

Technology Stack

AI/ML Layer:

  • NVIDIA NIM for LLM inference (llama-3.1-nemotron-nano-8B-v1)
  • NVIDIA NIM Embeddings for semantic search (nv-embedqa-e5-v5)
  • LangGraph for agentic workflow and multi-step reasoning
  • ChromaDB for vector storage and similarity search

Application Layer:

  • Streamlit for interactive web interface
  • Python 3.12 with async/await for concurrent operations
  • OpenAI-compatible client for NVIDIA NIM API integration

Infrastructure Layer:

  • Docker for containerization
  • Kubernetes manifests for orchestration
  • Terraform for AWS EKS infrastructure provisioning
  • AWS EKS for scalable deployment
  • AWS ECR for container registry

Building Process

Phase 1: Foundation (Days 1-2)

Started with the core agent logic using LangGraph. The key was designing a flexible agent that could:

  • Parse user queries
  • Create execution plans
  • Execute tools sequentially or in parallel
  • Synthesize final answers

Key Code Pattern:

class RAGAgent:
    async def plan(self, query: str) -> List[Step]:
        """Generate execution plan for complex queries"""

    async def execute_step(self, step: Step) -> Result:
        """Execute a single step with appropriate tool"""

    async def synthesize(self, results: List[Result]) -> Answer:
        """Combine results into coherent answer"""

Phase 2: NVIDIA NIM Integration (Day 3)

Integrated NVIDIA NIM for both inference and embeddings. Discovered that NVIDIA's API is OpenAI-compatible, which simplified integration:

from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.getenv("NIM_API_KEY")
)

The llama-3.1-nemotron-nano-8B-v1 model proved excellent for:

  • Fast inference (critical for multi-step reasoning)
  • Good reasoning capabilities
  • Cost-effective at scale

Phase 3: Retrieval Pipeline (Day 4)

Built the document processing and retrieval system:

  • Document ingestion with chunking (500 tokens, 50 overlap)
  • Embedding generation with NVIDIA NIM
  • Vector storage in ChromaDB
  • Similarity search with relevance scoring

Processed 683 documents across multiple formats, creating a knowledge base for testing.

Phase 4: Enhanced UI (Days 5-6)

Moved from basic interface to production-quality UI with:

  • Tabbed navigation (Chat, Documents, API Logs, Settings)
  • Real-time streaming responses
  • Document upload with progress feedback
  • API call tracking and visualization
  • Cost estimation dashboard

Phase 5: AWS Deployment Preparation (Day 7)

Created comprehensive infrastructure as code:

  • Terraform modules for VPC, EKS, node groups (CPU + GPU)
  • Kubernetes manifests for deployments, services, HPA, PVC
  • Docker containerization with multi-stage builds
  • Deployment documentation with step-by-step guides

💪 Challenges I Faced

Challenge 1: Model Name Format Issues ⚡

Problem: NVIDIA NIM's llama-3.1-nemotron-nano-8B-v1 requires exact model naming. Initially used incorrect formats causing API errors.

Solution:

  • Created diagnostic scripts to validate model names
  • Fixed environment variable format: meta/llama-3.1-nemotron-nano-8B-v1
  • Added validation in code to catch misconfigurations early

Learning: API integration requires precise attention to vendor-specific naming conventions.

Challenge 2: Embedding Input Type Mismatch 🔧

Problem: NVIDIA embedding API requires input_type="query" for search queries and input_type="passage" for documents. Missing this caused suboptimal retrieval.

Solution:

# For document embedding
response = await client.embeddings.create(
    model="nvidia/nv-embedqa-e5-v5",
    input=texts,
    input_type="passage"  # Critical parameter
)

# For query embedding
response = await client.embeddings.create(
    model="nvidia/nv-embedqa-e5-v5",
    input=query,
    input_type="query"  # Different parameter
)

Learning: Understanding model-specific requirements is crucial for optimal performance.

Challenge 3: PDF Processing Reliability 📄

Problem: Some PDF files failed to process due to encoding issues, corrupted files, or complex layouts.

Solution: Implemented robust error handling:

def extract_text_from_pdf(self, pdf_path: str) -> str:
    try:
        doc = fitz.open(pdf_path)
        text = ""
        for page in doc:
            text += page.get_text()
        return text or "Empty document"
    except Exception as e:
        logger.error(f"PDF processing failed: {e}")
        return f"[Error processing {pdf_path}]"

Learning: Always plan for edge cases in file processing—real-world data is messy.

Challenge 4: Async/Await Implementation 🔄

Problem: Mixing synchronous and asynchronous code caused event loop issues, especially with ChromaDB (sync) and NVIDIA NIM (async).

Solution:

  • Used asyncio.run() carefully in sync contexts
  • Created async wrappers for sync operations
  • Maintained consistent async patterns throughout

Learning: Choose async or sync early—mixing them adds complexity.

Challenge 5: Streamlit State Management 🎛️

Problem: Streamlit reruns the entire script on interaction, causing state loss and redundant API calls.

Solution: Leveraged st.session_state extensively:

if 'messages' not in st.session_state:
    st.session_state.messages = []

if 'agent' not in st.session_state:
    st.session_state.agent = initialize_agent()

Learning: Understanding framework-specific patterns is essential for smooth UX.

Challenge 6: Cost and Performance Optimization 💰

Problem: Initial implementation made excessive API calls, increasing costs and latency.

Solution:

  • Implemented response streaming for better perceived performance
  • Added caching for frequently accessed documents
  • Optimized chunk sizes (500 tokens) for balance between context and cost
  • Limited retrieval to top-k=5 most relevant documents

Learning: Production systems require careful resource management.

Challenge 7: Infrastructure Complexity 🏗️

Problem: AWS EKS deployment involves many moving parts—VPC, subnets, security groups, IAM roles, node groups, load balancers.

Solution:

  • Used Terraform modules for reproducible infrastructure
  • Created comprehensive deployment guide
  • Documented every step with troubleshooting tips
  • Estimated costs upfront (~$450-500/month)

Learning: Infrastructure as Code is essential for complex cloud deployments.

🎓 What I Learned

Technical Skills

  1. Agentic AI Design: Understanding how to build systems that can plan, reason, and use tools—not just pattern match
  2. NVIDIA NIM Platform: Deep experience with both inference and embedding APIs
  3. LangGraph Framework: Building complex multi-step workflows with state management
  4. Production RAG: Going beyond toy examples to handle real documents, errors, and edge cases
  5. AWS EKS: Deploying containerized applications at scale with Kubernetes
  6. Terraform: Infrastructure as Code for reproducible cloud environments

Architectural Lessons

  1. Modularity Matters: Separating concerns (agent logic, NIM clients, retrieval, UI) made debugging easier
  2. Observability is Key: API logging and reasoning transparency build trust and aid debugging
  3. Error Handling: Production systems must gracefully handle failures at every layer
  4. Documentation: Good docs are as important as good code—future you will thank present you

AI/ML Insights

  1. Smaller Models Can Be Powerful: llama-3.1-nemotron-nano-8B-v1 (8B parameters) performs remarkably well for RAG tasks
  2. Embeddings Quality Matters: Using proper input_type significantly improved retrieval accuracy
  3. Chunking Strategy: 500 tokens with 50-token overlap provides good balance
  4. Retrieval vs Generation: RAG is powerful when you need factual accuracy with source attribution

Soft Skills

  1. Systematic Debugging: Creating diagnostic scripts saved hours of trial-and-error
  2. Documentation as You Go: Maintaining bug fix logs and architecture docs prevented confusion
  3. User-Centric Design: Features like document viewer and API logs came from thinking about what users need
  4. Planning Before Coding: The comprehensive plan (HACKATHON_WINNING_PLAN.md) kept me focused

🚀 What's Next

If I had more time, I'd add:

  1. Advanced Analytics Dashboard: Visualize query patterns, retrieval accuracy, and system performance
  2. Multi-Document Comparison: Tool for side-by-side analysis of information across documents
  3. Collaborative Features: Allow multiple users to build shared knowledge bases
  4. Fine-Tuning: Customize embeddings for domain-specific documents
  5. Conversational Memory: Long-term memory across sessions for personalized interactions
  6. Export Capabilities: Generate reports from conversations and retrieved information

🏆 Why This Project Matters

This isn't just another chatbot or simple RAG demo. The NIM RAG Agent demonstrates:

✅ Transparency: Users see how AI arrives at conclusions ✅ Trustworthiness: Source citations with relevance scores ✅ Intelligence: Multi-step reasoning for complex queries ✅ Production-Ready: Scalable AWS infrastructure with monitoring ✅ Developer-Friendly: Clean architecture, comprehensive docs, easy to extend

In an era where AI systems are often criticized as "black boxes," this project shows that powerful AI can also be transparent and trustworthy.

🙏 Acknowledgments

  • NVIDIA for the incredible NIM platform—API compatibility made integration smooth
  • AWS for EKS documentation and generous compute resources
  • LangChain/LangGraph for excellent agent framework and examples
  • Streamlit for making beautiful UIs accessible to Python developers
  • The Open Source Community for ChromaDB, PyMuPDF, and countless other tools

📊 Project Stats

  • Total Development Time: ~60 hours over 7 days
  • Lines of Code: ~3,500 (excluding tests and configs)
  • Documents Processed: 683 documents
  • API Calls Handled: 1000+ during testing
  • Bugs Fixed: 3 critical bugs documented
  • Tests Written: 15+ unit and integration tests
  • Infrastructure Files: 10+ Terraform and Kubernetes manifests

🎯 Conclusion

Building the NIM RAG Agent taught me that the best AI systems are those that work with humans, not just for them. By making the agent's reasoning visible, providing source citations, and offering full transparency, we create AI that users can trust and understand.

The combination of NVIDIA's powerful language models and AWS's scalable infrastructure creates a foundation for building production-grade AI applications that can handle real-world complexity. This project proves that advanced agentic AI is not just research—it's practical, deployable, and ready to solve real problems.

I'm excited to see where this technology goes next, and I'm grateful for the opportunity to explore it through this hackathon.

Built with ❤️ for the NVIDIA NIM + AWS Hackathon

Built With

  • alb
  • asyncio
  • aws-cli
  • cloudwatch)-databases:-chromadb-(vector-database)-container/orchestration:-docker
  • ebs
  • ec2
  • ecr
  • fastapi-ai/ml:-nvidia-nim-(llama-3.1-nemotron-nano-8b-v1
  • hcl
  • html/css
  • iam
  • javascript
  • kubectl
  • kubernetes
  • langchain
  • languages:-python
  • markdown-frameworks:-langgraph
  • nv-embedqa-e5-v5)-cloud:-aws-(eks
  • openai-compatible-api-testing:-pytest
  • pytest-asyncio
  • pytest-cov-other:-git/github
  • streamlit
  • terraform-apis:-nvidia-nim-api
  • vpc
  • yaml
Share this project:

Updates