AgentHifazat — Multilingual AI Red-Team

Stress-test enterprise AI agents in Urdu, Arabic & English. Maps every failure to OWASP ASI 2026, compares vulnerable vs hardened builds, and exports audit-ready PDF reports for security teams.

Comment

Inspiration

Enterprise teams are shipping AI agents that handle money, PII, and regulated workflows — but most still rely on prompt-only guardrails tested only in English. In Pakistan and MENA markets, agents must work in Urdu, Roman Urdu, Arabic, and English. An agent that refuses a jailbreak in English often still leaks when the same attack is written in Urdu or Arabic.

I built AgentHifazat (Urdu: protection) to give security and engineering teams a repeatable way to stress-test agents before production — with country-specific attack packs and mapping to OWASP Agentic AI (ASI 2026).

What it does

AgentHifazat is a multilingual red-team harness for LangGraph-style agents:

  • Attack packs — Pakistan (50 attacks: Urdu / Roman Urdu / English), Global English, UAE/MENA (Arabic + English)
  • Live dashboard — run scans, view ASI heatmaps, inspect full multi-turn attack trajectories
  • Before/after comparison — same pack on vulnerable vs hardened FinBot (we measured 70% → 96% pass rate on Pakistan)
  • Judges — offline rules (CI-friendly), Groq LLM, or hybrid
  • Audit PDFs — one-click executive reports for compliance teams
  • CLI + REST API — compare, matrix, async scans, GitHub Actions CI

How I built it

  • Backend: Python, FastAPI, SQLite (run history + auth)
  • Frontend: React + Vite dashboard
  • Agent adapter: LangGraph FinBot demo (vulnerable + hardened)
  • Scoring: Rules engine + optional Groq LLM judge
  • Reports: PDF generator with ASI breakdown and failed-attack summaries
  • Packs: JSON/YAML attack definitions with PII patterns and regional context

Everything runs locally with Docker or uvicorn + npm run dev. Rules judge works fully offline — no API key required.

Challenges I ran into

  • Multilingual judging: Urdu and Roman Urdu attacks needed language-aware PII patterns, not just English regex.
  • Reproducibility: Every attack stores a full JSON trajectory so results can be replayed and re-scored with a different judge.
  • LLM reliability: When using Groq for judging, rate limits on free tier required a hybrid mode and offline rules as fallback for CI.
  • Tool output in demos: Building the demo agent taught me that LLMs often summarize away structured tool results — the fix was to surface tool output directly to the user.

Accomplishments that I'm proud of

  • Three working country packs with real ASI category coverage
  • End-to-end flow: scan → heatmap → trajectory → PDF in under 3 minutes
  • Measurable hardening story: 13 attacks fixed between vulnerable and hardened FinBot on the Pakistan pack
  • Full test suite (pytest) and GitHub Actions CI

What I learned

Red-teaming AI agents is not the same as testing a chatbot. You need structured attack catalogs, category mapping (ASI), and evidence trails — not ad-hoc prompt tries. Multilingual testing surfaces failures that monolingual QA completely misses.

What's next

  • Live demo deploy on Render
  • HealthBot demo agent + Hindi/India pack
  • Hugging Face Space for public trials
  • Outreach to Pakistan software houses and MENA fintech teams

AI disclosure

  • Groq LLM is used optionally for hybrid/LLM judge modes and live FinBot targets.
  • Development: AI coding assistants helped with boilerplate and debugging; architecture, attack pack design, demo script, and project narrative are my own work.
  • Demo video: Recorded and narrated by me (not AI-generated voice).

Built With

Share this project:

Updates