Eagle - Data Infrastructure and Security Monitoring

Eagle is a real-time data platform for banks, enabling seamless data movement, real-time fraud detection, and AI-powered insights with LLMs for secure, efficient decision-making and proactive alerts.

Comment

👉🏾 Full Documentation & Code

Inspiration

🦅 Eagle is a cutting-edge real-time data platform designed for the banking sector, offering seamless data movement, real-time fraud detection, and AI-powered insights using LLMs. It empowers banks to make secure, efficient decisions while proactively addressing threats and anomalies.

Eagle was conceived to tackle the challenges posed by massive data volumes. With billions of transactions generated daily, having a robust system to facilitate the seamless data migration from its point of origin to its final destination is critical. Eagle ensures data flows efficiently and securely, enabling organizations to focus on leveraging insights and enhancing operations without worrying about infrastructure bottlenecks.

What it does

🦅 Eagle is broken down into 3 major components:

Part 1: Data Integration

Eagle is a powerful data infrastructure solution designed to handle real-time data ingestion from banking applications and APIs. It integrates with Azure Event Hub’s Transaction Namespace (similar to Apache Kafka), where messages are temporarily queued and stored for up to one hour before being forwarded to their final destination.

The data ingestion process is facilitated by a microservice architecture using Azure Functions with a Timer trigger. This acts as a producer, efficiently sending data at predefined intervals to Azure Event Hub to ensure consistent and reliable data flow.

Data from Azure Event Hub is consumed through two key pathways:

  1. Azure Stream Analytics: Aggregates and processes real-time data streams, inserting the results into an Azure SQL Database. This enables AI engineers and business users to access actionable, structured insights.
  2. Python Integration: A Python-based solution consumes data directly from Azure Event Hub and stores it in an Azure Storage Account, providing a flexible alternative for data storage and downstream processing.

Part 2: Security, Monitoring, and Alerts

Eagle incorporates advanced security and monitoring capabilities to safeguard sensitive data and detect suspicious activities. The consumer script intelligently routes data to two storage accounts: one for standard transactions and another dedicated to suspicious transactions.

Suspicious Transactions are flagged using the following criteria:

  • Transactions involving individuals from countries with international sanctions, such as "PRK," "IRN," "SYR," and "CUB."
  • Dormant accounts that suddenly initiate large fund transfers. Flagged transactions are automatically routed to the Suspicious Transactions storage account. An Azure Function with an Event-based Blob Trigger monitors this account and sends immediate alerts to the security and anti-fraud team for further investigation. This ensures prompt action against potential threats.

Part 3: Q&A with LLM and Deployment

Eagle leverages the Azure OpenAI API to build a Retrieval-Augmented Generation (RAG) model on transaction data stored in the Azure SQL Database. This intelligent system allows business users to query their data conversationally and receive insights tailored to their needs. You can also generate report and dashboard based of the question asked.

The solution is deployed using Streamlit and hosted on Azure WebApp, providing an intuitive and accessible interface. With Eagle, business users can effortlessly explore their data, gain deeper insights, and make data-driven decisions in real time.

How we built it

Eagle was developed using a combination of Azure Data Cloud technologies(Azure SQL Database, Azure Storage Account - ADLS, Azure Function - Timer & Event, Azure WebApp, Azure Event Hub, Azure Stream Analytics, and Python, leveraging the power of modern cloud infrastructure for scalability and efficiency. The development process was streamlined with the help of VS Code and GitHub Copilot, which accelerated coding and problem-solving.

Continuous improvement was a core focus, involving iterative debugging and optimization of multiple scripts to ensure seamless functionality, robust performance, and adaptability to real-world challenges. By combining cutting-edge tools with agile development practices, Eagle was crafted to deliver a reliable and scalable solution for managing real-time data and fraud detection in the banking sector.

Challenges we ran into

  • Azure Function Deployment Challenges: A significant challenge was encountered when deploying Azure Functions, particularly in transferring environment variables from the local VS Code development environment to the Azure Function cloud. Variables defined in .env and local.settings.json files were not automatically recognized in the Azure Function during deployment. This required manual configuration of variables within the Azure Function App’s settings in the Azure portal. The expectation was for the .env file to be packaged and deployed alongside the Function_App.py script, streamlining the deployment process. Automating this would enhance the efficiency and reliability of deployments.

  • Azure Stream Analytics Integration with Azure SQL Database: Integrating Azure Stream Analytics with Azure SQL Database required careful preparation, including manually creating the target table in the database with the exact data types matching the producer's output. This step proved challenging as we expected Azure Stream Analytics to automatically create the necessary table in Azure SQL Database when configured as an output. Moreover, identifying and aligning the correct data types from the producer added complexity to the process.

Enabling Azure Stream Analytics to create tables directly in Azure SQL Database would be a transformative enhancement, similar to the functionality available in Power BI workspaces for Stream Datasets. This capability would significantly simplify integration workflows and reduce setup time.

Accomplishments that we're proud of

We’re thrilled with the seamless integration and successful implementation of all key components in the Eagle system. Key achievements include:

  • Azure Function Timer Trigger: Functioning flawlessly to ensure data is ingested and sent to Azure Event Hub at precise intervals.
  • Azure Blob Trigger: Performing as expected, enabling real-time monitoring and alerting for suspicious transactions.
  • Azure Stream Analytics Integration: Successfully processing and aggregating data streams, with smooth insertion into Azure SQL Database for further analysis.
  • LLM Deployment: Successfully deploying the AI-powered LLM solution to Azure WebApp via Streamlit, providing business users with a robust, interactive interface for querying and analyzing data.

These milestones demonstrate the strength of the solution and our ability to overcome complex challenges in real-time data processing and AI integration.

What we learned

We gained valuable insights into integrating Azure Stream Analytics with SQL databases more effectively, including best practices for aligning data types and optimizing performance. Additionally, we refined our approach to Azure Function deployments, learning how to streamline the process and manage environment variables efficiently for a smoother transition from local development to the cloud.

What's next for Eagle - Data Infrastructure and Security Monitoring

The next phase for 🦅 Eagle involves making it production-ready by integrating with Azure Kubernetes Service (AKS), enabling scalability and robustness for enterprise-level deployments. This will allow Eagle to efficiently handle large volumes of data, support high availability, and provide enhanced flexibility in managing resources at scale, ensuring the solution can meet the demands of modern, data-intensive banking environments.

Built With

  • azure
  • azurefunctions
  • azuresqldatabase
  • azurestorageaccount
  • azurewebapp
  • copilot
  • eventhub
  • powershell
  • python
  • slack
  • streamanalytics
  • streamlit
  • vscode
Share this project:

Updates