DiffSense

Every ‘simple’ code change risks breaking your app. DiffSense’s AI catches breakages early—and lets you query your entire code history with an LLM powered by retrieval-augmented generation (RAG).

Comment

DiffSense

AI-Powered Code Change Intelligence & Breaking Change Detection

Turning code deployment from Russian roulette into predictable science

Inspiration

Every developer has that one commit that haunts their dreams. A "simple" function rename that crashes the mobile app. A "harmless" API update that locks out millions of users. A database schema change that wipes data forever.

The inspiration for DiffSense came from watching brilliant engineering teams repeatedly fall into the same trap: invisible breaking changes. We realized that these disasters aren't random—they're predictable. The connections between code changes and their cascading effects follow patterns that human eyes miss but AI can see.

What it does

DiffSense is like having a crystal ball for your code changes. It's an AI-powered platform that:

Predictive Breaking Change Detection

  • Analyzes every code change using advanced ML models
  • Predicts ripple effects across your entire codebase
  • Provides impact severity scoring and affected user estimates

Intelligent Context Analysis

  • Uses RAG (Retrieval-Augmented Generation) to understand your codebase
  • Leverages Claude AI for natural language explanations
  • Performs deep AST analysis and pattern detection
  • Maintains historical context of previous breaking changes

Smart Recommendations

  • Generates actionable suggestions to prevent disasters
  • Provides migration strategies for necessary breaking changes
  • Prioritizes fixes by business impact and technical risk

Real-time Integration

  • GitHub OAuth integration for seamless repository access
  • WebSocket support for real-time change monitoring
  • RESTful API for integration with existing CI/CD pipelines
  • Vector embeddings for semantic code similarity analysis

How we built it

Architecture

We built DiffSense using a modern, AI-first architecture:

Frontend: React with Vite, TailwindCSS for beautiful, responsive UI Backend: FastAPI with Python for high-performance API endpoints AI/ML Stack:

  • Claude AI for natural language understanding
  • CodeBERT for code semantic analysis
  • SentenceTransformers for vector embeddings
  • Custom ML models for breaking change prediction

Data Layer: SQLite for development with vector caching for embeddings Infrastructure: Layered architecture supporting future microservices migration

Key Technical Innovations

  1. Semantic Code Analysis: Combined AST parsing with ML embeddings
  2. Intelligent RAG System: Context-aware retrieval for better AI responses
  3. Predictive Breaking Change Models: Custom ML pipeline trained on git history patterns
  4. Real-time Change Tracking: WebSocket-based live monitoring system

Development Process

  • AI-first design philosophy from day one
  • Extensive testing with real-world repositories
  • User-centered design for developer experience

Challenges we ran into

Technical Challenges

  1. Scale of Code Analysis: Analyzing entire repositories efficiently without overwhelming resources
  2. Context Window Limitations: Working within AI model token limits while maintaining comprehensive analysis
  3. False Positive Management: Balancing sensitivity with practical usability
  4. Real-time Performance: Delivering instant feedback without sacrificing accuracy

Integration Challenges

  1. GitHub API Rate Limits: Efficiently managing API calls for large repositories
  2. Cross-platform Compatibility: Supporting diverse development environments
  3. Security & Privacy: Handling sensitive code data with enterprise-grade security
  4. Performance Optimization: Maintaining speed while processing complex codebases

Accomplishments that we're proud of

Technical Achievements

  • Built a working AI system that can actually predict breaking changes
  • Utilized a breaking change detector to power our web app
  • Fine tuned a RAG model to smart prompt a Claude API

Innovation

  • First-of-its-kind predictive breaking change detection
  • Novel combination of static analysis, ML, and AI reasoning
  • Beautiful, intuitive UX that makes complex AI accessible

What we learned

Technical Insights

  1. AI works best with structure: Combining traditional static analysis with AI gives better results than pure AI
  2. Context is everything: The quality of our RAG system directly impacts AI accuracy
  3. User feedback is gold: Real developer usage patterns taught us what actually matters

What's next for DiffSense

  • VS Code Extension with inline risk alerts and code search
  • Fine-tune models on commit-level breaking changes
  • Improve code embeddings & retrieval quality (RAG)
  • Add context retention for multi-turn queries
  • Scale backend infrastructure for faster, larger repo support

The Future is Predictable

DiffSense represents a fundamental shift in how we think about code changes. Instead of crossing our fingers and hoping for the best, we're moving toward a world where every change is analyzed, understood, and deployed with confidence.

We're not just preventing bugs. We're preventing disasters.

The future of software isn't about writing perfect code—it's about knowing exactly what your imperfect code will do before it's too late.

Welcome to the future of predictable code deployment. Welcome to DiffSense.

Built With

Share this project:

Updates