SentientGate: AI-Powered Runtime Security

SentientGate is an AI-powered runtime security fabric for microservices, pairing ultra-fast gateway blocking, Kafka event streaming, and local LLM heuristics to neutralize active threats in real time.

Comment

Inspiration

⚡ The Problem: The Security vs. Speed Dilemma

In modern cloud architectures, microservices process millions of requests every minute. Protecting them today forces engineers into a painful compromise:

  • ❌ Traditional WAFs & Gateways are static: They depend on rigid regex rules and basic IP rate limits. They fail against low-and-slow brute-force, API credential stuffing, and evolving zero-day attacks.
  • ❌ Cloud AI solutions are slow & costly: Third-party AI security platforms charge steep SaaS fees, expose sensitive company data to external clouds, and add 200–500ms of latency directly into every user request.

Our Question: Why must developers sacrifice sub-millisecond API performance just to get intelligent, adaptive security?

💡 The Solution: SentientGate

We built SentientGate to eliminate this compromise. It is a self-hosted, cloud-native security fabric that sits directly in front of microservices. It forwards legitimate traffic with <1ms overhead, while asynchronously streaming telemetry to an out-of-band security brain powered by heuristic threat rules and privacy-first local LLMs (via Ollama) to neutralize attacks in real time.

What it does

🚀 The 10-Second Summary

SentientGate observes live API traffic, allows clean requests instantly, and inspects suspicious actors in the background. When an anomaly is detected, it automatically places a temporary, auto-expiring block on the attacker via Redis.

🏛️ System Architecture Diagram

SentientGate Architecture

🔗 View high-resolution Architectural Diagram on GitHub

🔄 End-to-End Request & Threat Lifecycle

SentientGate Sequence Diagram

🔗 View high-resolution Sequence Diagram on GitHub

🔍 How It Works (Step-by-Step)

  1. Sub-Millisecond Edge Enforcement (<1ms): Every incoming request is verified against an in-memory Redis blacklist. If clean, the request is forwarded to backend microservices with zero perceptible delay.
  2. Tamper-Proof Visitor Tracking: Instead of trusting plain client headers or IP addresses (which attackers easily spoof), the gateway issues an HMAC-SHA256 signed HTTP-only cookie (SENTINEL_VISITOR_ID) to persistently track bad actors across sessions.
  3. Out-of-Band Threat Analysis: All request audits stream asynchronously into Apache Kafka. If an endpoint responds with error codes (4xx or 5xx), a security event is triggered without delaying the client's HTTP response.
  4. Layered Detection Engine (MCP Service): The security engine pulls the client's recent history via high-speed gRPC and checks:
    • Burst Traffic: Detects volumetric DDoS spikes and automated fuzzing.
    • Pattern Matching: Spots normalized SQL Injection (UNION SELECT), XSS, and Path Traversal (../).
    • Sensitive Path Probing: Catches unauthorized scans hitting .env, /actuator, or /admin.
    • Rate-Limit Escalation: Converts repeat 429 Too Many Requests violations into temporary blocks.
  5. Local AI Anomaly Scoring (Ollama): If an attack is subtle and bypasses heuristic rules, the request metadata is analyzed by a local Large Language Model (e.g., Llama 3 / Qwen 2.5). The LLM checks behavioural anomalies (rapid IP shifting, abnormal endpoint traversal) and scores the threat.
  6. Smart TTL-Based Blocking: Instead of permanent IP bans that cause accidental collateral damage, SentientGate applies dynamic Time-To-Live (TTL) blocks in Redis (e.g., 5 min for minor violations, 60 min for severe attacks). Systems recover automatically without manual admin cleanup.
  7. Real-Time Operator Dashboard: Security teams monitor live metrics, inspect an interactive visual architecture pipeline, review audit ledgers, and manage active blocks with live countdown timers on a sleek React dashboard.

How we built it

SentientGate was built from the ground up as an event-driven, production-ready microservices platform:

Component Technology Primary Role
API Gateway Java 21, Spring Cloud Gateway (WebFlux / Netty) Non-blocking edge proxy, HMAC visitor signing, Redis blacklist enforcement
Security Brain (MCP) Spring Boot 3, Isolated Thread Pools, Resilience4j Multi-strategy threat heuristic evaluation, rate-limit cooling, AI orchestration
AI Service Spring WebFlux, Reactive WebClient, Ollama Local LLM inference for contextual behavioral anomaly detection
Logging & Telemetry Spring Boot, PostgreSQL, Redis, gRPC / Protobuf Dual persistence (Postgres audit + 10m Redis cache) and high-speed binary log queries
Service Registry Spring Cloud Netflix Eureka Dynamic microservice discovery and client-side load balancing
Security Dashboard React 18, TypeScript, Vite, Tailwind CSS Real-time traffic analytics, interactive pipeline flow, and threat drawer
Cloud & DevOps Docker, Kubernetes (Helm), Terraform, GitHub Actions Complete IaC (AWS EKS), container orchestration, and automated CI/CD quality gates

Challenges we ran into

1. The "Latency vs. Intelligence" Paradox

  • The Problem: Running LLM inference (100–500ms) or deep packet analysis directly in the HTTP request path destroys API throughput.
  • Our Solution: We decoupled analysis entirely through an Out-of-Band (OOB) event-driven architecture. Traffic is forwarded immediately; telemetry is published asynchronously to Kafka. If an anomaly is detected, the attacker's ID is written to Redis, blocking their subsequent requests in <1ms.

2. Thread Starvation During Attack Spikes

  • The Problem: Under heavy attack traffic, slow AI inference calls threatened to exhaust worker threads, stalling Kafka listeners and delaying fast heuristic checks.
  • Our Solution: We architected strictly isolated thread pools:
    • mcp-analysis-*: Dedicated exclusively to fast heuristic rules.
    • mcp-ai-*: Dedicated exclusively to Ollama LLM requests. Combined with a CallerRunsPolicy and Redis deduplication guards, this ensured high-throughput backpressure without message loss.

3. Tracking Bad Actors Without Trusting Them

  • The Problem: Attackers rotate user-agent headers, switch IP addresses, or clear browser cookies to bypass rate limits.
  • Our Solution: We engineered a cryptographic HMAC-SHA256 signature into the visitor cookie. The gateway cryptographically validates the token on every request, making it impossible for attackers to forge or tamper with their tracking identity.

4. Preventing AI Hallucinations in Automated Security

  • The Problem: Large Language Models can output arbitrary, unpredictable prose that cannot be parsed reliably by automated backend services.
  • Our Solution: We designed strict JSON output system prompts, coupled with defensive score parsing, timeouts, and Resilience4j circuit breakers. If the AI model times out or stutters, the system fails safely and relies on heuristic rules.

Accomplishments that we're proud of

  • ⚡ <1ms Edge Latency: Validated sub-millisecond Redis blacklist checks at the gateway, introducing virtually zero delay to clean user traffic.
  • 🔒 100% On-Premise Data Privacy: Security telemetry and payloads never leave the private infrastructure — AI inference runs entirely locally via Ollama.
  • 🧩 Extensible Strategy Pattern: Designed a modular threat detection engine in Java that allows new detection rules (e.g., custom regex, path filters) to be added with minimal code.
  • 🚀 Binary gRPC Performance: Replaced JSON/REST with binary Protobuf gRPC streaming between internal services, slashing internal network serialization overhead by over 60%.
  • 📦 True Production Readiness: Delivered a complete end-to-end cloud platform, including full Terraform AWS EKS provisioning, production Kubernetes Helm charts, and automated GitHub Actions CI/CD pipelines.

What we learned

  • Reactive Microservices at Scale: Deepened our expertise in Spring WebFlux, Project Reactor, and Netty event loops for high-concurrency, non-blocking I/O.
  • Resilience & Fault Isolation: Learned how to safeguard critical microservice pipelines using thread-pool bulkheads, circuit breakers, and Redis deduplication guards.
  • Practical AI Engineering: Discovered how to bridge the gap between non-deterministic local LLMs and high-reliability backend systems through disciplined prompt engineering and fallbacks.
  • Full-Lifecycle DevOps: Gained hands-on experience unifying infrastructure provisioning (Terraform), container orchestration (Kubernetes/Helm), and continuous integration into a cohesive developer workflow.

What's next for SentientGate: AI-Powered Runtime Security

  • 🐧 Kernel-Level eBPF/XDP Enforcement: Push Redis blacklist lookups directly into the Linux kernel via eBPF to drop malicious packets at wire speed before they reach user space.
  • 🧠 RAG-Powered Threat Memory: Add vector search (pgvector) to cross-reference incoming attack behavior with multi-day distributed scanning campaigns.
  • 🍯 Automated Dynamic Honeytokens: Automatically inject decoy API endpoints and fake credentials into responses to instantly trap and fingerprint automated bots.
  • 🌐 Service Mesh Integration (Envoy / Istio): Package SentientGate's edge filter as an Envoy WebAssembly (Wasm) plugin to secure multi-cloud service meshes out of the box.

Built With

  • amazon-eks
  • apache-kafka
  • docker
  • github-actions
  • grpc
  • helm
  • java-21
  • kubernetes
  • llama-3
  • ollama
  • openfeign
  • postgresql
  • project-reactor
  • protobuf
  • react-18
  • redis
  • resilience4j
  • spring-boot-3
  • spring-cloud-eureka
  • spring-cloud-gateway
  • spring-webflux
  • tailwindcss
  • terraform
  • typescript
  • vite
Share this project:

Updates

Submission history