HIDE - A Unified GNN Framework for Multi-Paradigm Anomaly Detection

HIDE (HIdden Dynamics Engine) is a domain-agnostic, strategy-driven framework that uses a single, powerful, and configurable Graph Neural Network (GNN) to detect hidden, anomalous behaviors across diverse, real-world systems.

This project's core innovation is a unified model architecture based on GraphSAGE that can be dynamically configured to perform unsupervised, supervised, and semi-supervised learning. This demonstrates a novel and flexible approach to anomaly detection, moving beyond single-purpose models to a true, reusable AI engine.

---

Table of Contents

1. The Core Innovation: A Unified, Multi-Paradigm GNN
2. Implemented Domain Strategies
3. Project Structure
4. Getting Started
5. How to Run the Framework
   • Step 1: Train a Model
   • Step 2: Run Prediction
6. Docker Execution (Recommended)

---

The Core Innovation: A Unified, Multi-Paradigm GNN

HIDE's architecture is built to showcase a single, powerful idea: that a well-designed Graph Neural Network can be the universal tool for finding anomalies, regardless of the learning paradigm.

• Unified Data Representation: The framework treats all problems as a graph. For tabular data, it constructs a "similarity graph"; for transactional data, it builds a relational graph; for native graph data, it uses the provided structure.
• A Single, Powerful Strategy: All domain logic is encapsulated in the UnifiedGnnStrategy. This central module contains a GNN model with adaptable "heads" that are activated based on the learning_paradigm set in the domain's YAML file.
  • Unsupervised Mode (Graph Autoencoder): For domains with no labels, the GNN learns to reconstruct node features. The reconstruction error is the anomaly score.
  • Supervised & Semi-Supervised Mode (Classifier): For domains with full or partial labels, the GNN is trained as a classifier. The anomaly score is the predicted probability of the "anomalous" class (e.g., fraud or illicit).
• Train/Predict Workflow: The framework follows a professional MLOps pattern, separating model training (train.py) from on-demand prediction (score.py).

---

Implemented Domain Strategies

HIDE demonstrates its power and flexibility across four high-impact domains, each utilizing a different learning paradigm with the same underlying model architecture.

1. Dark Pools (Unsupervised)

• Objective: Identify anomalous trading venues from aggregate FINRA ATS data.
• Method: A Graph Autoencoder is used to find venues with the most unusual statistical profiles and neighborhood structures—anomalies that simple heuristics would miss.

2. Supply Chain (Supervised)

• Objective: Identify fraudulent suppliers within a real-world supply chain dataset.
• Method: The GNN is trained as a Supervised Classifier to predict which suppliers are fraudulent based on their transactional and relational features.

3. Blockchain AML (Semi-Supervised)

• Objective: Classify "unknown" transactions in the Elliptic Bitcoin dataset as potentially illicit.
• Method: The GNN is trained as a Semi-Supervised Classifier, leveraging the entire graph structure to propagate knowledge from the few labeled nodes to the vast unlabeled majority.

4. Energy Theft (Supervised)

• Objective: Predict fraudulent activity using a real-world financial transaction dataset as a powerful proxy for labeled energy billing data.
• Method: The GNN is trained as a Supervised Classifier on the tabular data to demonstrate its applicability beyond native graph problems.

---

Getting Started

Prerequisites

• Python 3.9+
• Docker (Strongly Recommended for easiest setup)

Data Setup

• Dark Pools: Place your finra-ats-*-nms.xlsx files inside a data/finra_quarterly/ folder.
• Energy Theft / Supply Chain: These datasets will be downloaded automatically by the script. Fallback local files can be placed in data/energy/ and data/supply_chain/.
• Blockchain AML: Manual download required.
  1. Download from Kaggle: Elliptic Data Set.
  2. Create a folder: data/elliptic/.
  3. Place the three CSV files inside it.

---

How to Run the Framework

The workflow is a two-step process: first you train a model for a specific domain, then you use that model to run predictions.

Step 1: Train a Model

Open your terminal in the project root directory. First, set up your environment and install dependencies.

# Create & activate a virtual environment
python -m venv venv
.\venv\Scripts\Activate.ps1
# Install all required packages
pip install -r requirements.txt

Built With

• docker
• html5
• python
• yaml