Jarvis AI

Next-gen code intelligence on GCP + Elasticsearch: Advanced RAG, BM25+semantic hybrid search, Gemini reranking, infinite memory via summarization, multimodal debugging & smart assistance.

Comment

Inspiration

As developers, we constantly face the challenge of understanding large, unfamiliar codebases. Traditional search tools lack context, and LLMs lose track of long conversations. We envisioned Jarvis AI - an intelligent code assistant that truly understands your entire repository and maintains infinite conversation memory through smart summarization.

What it does

Jarvis AI is a next-generation code intelligence platform that:

  • Ingests entire GitHub repositories and indexes them using Elasticsearch with both BM25 (keyword) and semantic (vector) search
  • Provides RAG-powered multimodal chat with full codebase context using hybrid retrieval + Gemini reranking for maximum accuracy
  • Smart GitHub sync - automatically fetches and ingests repositories from GitHub URLs with full codebase parsing
  • Maintains infinite chat memory via automatic summarization every 5 Q&A pairs with vector-indexed storage
  • Offers multimodal debugging - upload error screenshots and get AI-powered solutions with relevant code references
  • Tracks repository status - monitor ingestion progress, embedding status, and indexing completion
  • Delivers contextual Q&A - ask about any function, class, or logic pattern across your entire codebase
  • External search integration - DuckDuckGo web search for questions beyond the codebase
  • Real-time streaming responses - see AI answers appear live with character-by-character typing effect

How we built it

Tech Stack:

  • Frontend: Django templates with Tailwind CSS for responsive UI
  • Backend: Django 5.2 with async views for real-time chat
  • Search: Elasticsearch for hybrid BM25 + semantic vector search
  • AI: Google Gemini Pro for answer generation, reranking, and code analysis
  • Storage: Google Cloud Storage for media files, Cloud Run for deployment
  • Database: PostgreSQL for relational data, SQLite for dev/demo

Architecture:

  1. Repository ingestion: Parse files → Chunk code → Generate embeddings → Index in Elasticsearch
  2. Query pipeline: User question → Hybrid search (BM25 + semantic) → Gemini reranks top results → Generate answer with context
  3. Memory management: Track token usage → Auto-summarize when limit reached → Store summaries → Include in future context
  4. Multimodal debugging: Upload screenshot → Extract error text → Search codebase → Suggest fixes with code snippets

Challenges we ran into

  • Context window limits: Solved by implementing automatic summarization that condenses chat history while preserving key information
  • Hybrid search tuning: Balancing BM25 keyword matching with semantic similarity required extensive experimentation with weight parameters
  • Reranking optimization: Initially used simple cosine similarity, upgraded to Gemini-powered reranking for 40% better relevance
  • Deployment complexity: Configured Cloud Run with Elasticsearch VM, Google Cloud Storage, and proper environment variable management
  • Django 5.x compatibility: make_random_password() was removed - migrated to Python's secrets module for guest user generation

Accomplishments that we're proud of

✅ Built a production-ready RAG system with advanced hybrid search
✅ Implemented infinite chat memory via intelligent summarization
✅ Achieved seamless multimodal debugging with image upload support
✅ Deployed on Google Cloud Platform with Elasticsearch integration
✅ Created an intuitive UI that makes complex AI interactions simple

What we learned

  • Advanced RAG patterns: Hybrid retrieval significantly outperforms single-method search
  • Gemini API capabilities: Reranking and structured output features are game-changers
  • Cloud architecture: Properly configuring Cloud Run with external services (Elasticsearch) requires careful environment management
  • UX design: Making AI explanations clear and actionable is as important as the AI itself

What's next for Jarvis AI

  • Multi-repo support: Work across multiple related repositories simultaneously
  • Team collaboration: Shared chat sessions with annotations and bookmarks
  • VSCode extension: Bring Jarvis directly into the IDE workflow
  • GraphRAG Enhancement: Combine traditional RAG with knowledge graph traversal for deeper semantic understanding of code dependencies

Built With

Share this project:

Updates