Inspiration
It all started with a late-night conversation about the January 2025 Barclays banking outage in the UK.
Thousands of customers experienced disruption accessing banking services and making payments. As two first-year students, we found ourselves asking one question:
How can one software failure affect so many people, and why is finding its root cause so difficult?
We started researching how engineers investigate production incidents. We discovered that monitoring tools can identify failures, GitHub records code changes, Jira tracks requirements, and APM tools reveal application behaviour.
The information exists. But the connections between these pieces are often difficult to reconstruct.
That insight inspired BlastTrace: an AI-powered platform connecting software development history with production incidents to help engineers predict risks and investigate failures.
For the PayPal AI Hackathon, we explored another opportunity: making specialized AI investigations accessible on demand through PayPal, rather than requiring every customer to purchase an entire platform.
What it does
BlastTrace helps developers understand what might break before deployment—and investigate what actually broke afterward.
It combines two interconnected engines.
1. Software Factory — Proactive Intelligence
Before deployment, BlastTrace:
- Analyzes Jira requirements and relevant code components.
- Maps dependencies using FalkorDB and ChromaDB.
- Predicts the potential blast radius of software changes.
- Generates AI-assisted test and risk-assessment blueprints.
- Checks proposed changes against project-specific rules.
Developers can identify potential problems before those changes reach production.
2. Incident Investigation — Reactive Intelligence
When an incident occurs, BlastTrace:
- Receives monitoring alerts through Datadog.
- Connects error evidence with application dependencies, deployments and GitHub commits.
- Uses AI and graph-based investigation to rank likely root causes.
- Displays supporting code changes, technical evidence and confidence scores.
- Allows engineers to validate findings and provide feedback for future investigations.
Rather than giving developers an unexplained AI answer, BlastTrace helps them understand which change may have caused the failure and why.
PayPal-Powered, On-Demand Investigations
For the PayPal AI Hackathon, we extended this concept with a Sandbox payment integration.
The purchasing model offers two options:
- Deep AI Root Cause Analysis: Purchase an individual investigation.
- Software Factory Only: Choose a lower-priced proactive analysis package without purchasing full RCA functionality.
The backend includes a score-based eligibility check for paid RCA and a PayPal capture-verification step before delivery through the paid-result endpoint.
This creates two potential opportunities for PayPal: its engineering teams could use BlastTrace internally, while external developers could purchase AI-powered investigations through PayPal.
Our goal is simple: Give developers the capability they need without forcing an all-or-nothing purchase.
How we built it
We developed BlastTrace using Python and FastAPI, with a hybrid knowledge-graph and AI architecture.
FalkorDB is the foundation of our investigation engine. It connects Jira tickets, GitHub commits, code components, services, deployments and incidents into a unified knowledge graph.
Using multi-hop Cypher queries, BlastTrace traces relationships between production symptoms and potentially responsible code changes.
We combined this with:
- ChromaDB: Semantic code search, change similarity and historical incident context.
- Groq-powered AI: Test-blueprint generation and root-cause analysis.
- GitHub: Actual commit history, code diffs and project-specific rulebooks.
- Jira: Requirements, bug reports and development context.
- Datadog: Real monitoring webhook integration.
- APM simulation: Representative application traces for controlled testing.
- PayPal Sandbox: Order creation, payment capture verification and paid-result access in the hackathon integration.
Both engines share connected development evidence, allowing information collected before deployment to support later incident investigations.
For testing, we forked Salesforce's open-source E-Bikes Lightning Web Components application and deliberately introduced two bugs.
We committed those changes to GitHub and used the resulting incidents to evaluate whether BlastTrace could trace real code changes through the investigation pipeline.
The bugs were controlled. The Git history was real. The investigation ran through our actual graph and AI analysis components.
Challenges we ran into
Connecting multiple systems
Integrating Jira, GitHub, Datadog, graph queries, semantic search and AI reasoning was significantly more difficult than connecting individual APIs. Each source represented a different part of the software lifecycle.
Making the AI evidence-aware
We didn't want an AI system that simply guessed from error messages. We needed it to retrieve relevant graph relationships and code changes before generating conclusions.
Debugging our own debugging platform
One of our most memorable challenges took almost an entire day to diagnose. BlastTrace wasn't producing the expected results, and we suspected everything from graph relationships to retrieval logic.
The actual problem? We had accidentally been clearing the very codebase data BlastTrace needed for analysis.
We were building a root-cause investigation system while struggling to identify the root cause of our own failure!
Designing trustworthy payment access
Adding PayPal introduced a different type of engineering challenge: ensuring that viewing a successful checkout page could not, by itself, unlock a paid result.
We added server-side payment verification and tested eligibility and unauthorized-delivery scenarios. A complete live Sandbox transaction and deployment-grade access protection remain important final verification steps.
Accomplishments that we're proud of
Our biggest accomplishment is turning a late-night question into a working AI engineering platform.
We're especially proud of:
- Building two connected engines for proactive risk analysis and reactive investigation.
- Implementing FalkorDB-based relationships between requirements, code, services, deployments and incidents.
- Combining graph traversal, semantic retrieval and AI-generated analysis.
- Connecting real Datadog monitoring events with the investigation workflow.
- Demonstrating investigations using deliberately introduced bugs and real Git commits.
- Supporting evidence-backed root-cause rankings and engineer feedback.
- Building an interactive interface that makes complex technical relationships easier to explore.
- Extending the platform toward flexible, PayPal-powered AI investigation purchases.
What began as a project we weren't sure we could build became a functioning system with its own architecture, frontend and product vision.
What we learned
The most important lesson was that better AI reasoning starts with better-connected evidence.
An AI model is far more useful when it can examine code history, dependencies, deployment context and monitoring evidence together, rather than interpreting an isolated error message.
We also learned that real-world integrations require careful handling of unreliable data, missing relationships, inconsistent API responses and operational failures.
Working on BlastTrace taught us how to combine graph databases, semantic retrieval, AI models and payment APIs into a larger product workflow.
Most importantly, we learned that ambitious engineering projects become manageable when teammates keep experimenting, debugging and improving together.
What's next for BlastTrace
Our next priorities are to:
- Validate the complete PayPal Sandbox purchase-to-delivery journey and strengthen production authentication and access controls.
- Calibrate RCA confidence scores against more independently verified incidents.
- Expand testing beyond controlled bugs into additional real-world failure scenarios.
- Improve Datadog, APM and GitHub evidence correlation.
- Strengthen human feedback and persistent incident knowledge.
- Explore pay-per-investigation, lower-cost proactive analysis and flexible service packages.
- Evaluate how BlastTrace could support large engineering organizations, including PayPal's own development teams.
Our long-term vision is to make AI-powered software reliability intelligence more accessible, explainable and flexible to purchase.
BlastTrace — Predict the risk. Trace the cause. Pay for what you need.
