Inspiration
AI agents are shipping fast in Urdu and Roman Urdu markets, but most teams have no way to test whether a bot will leak PII, give unsafe advice, or break under prompt injection before real users hit it. I built AgentHifazat to make that failure visible early — especially for low-resource languages that most red-team tools ignore.
What it does
AgentHifazat is a multilingual agent red-teaming workbench. It runs structured attack packs (Urdu, Roman Urdu, English) against a target agent, logs full trajectories, scores pass/fail with rule-based judges, and exports audit-ready JSON reports. A vulnerable vs hardened FinBot demo shows measurable improvement (e.g. ~68% → ~96% pass rate on the Pakistan pack).
How I built it
- OpenAI tools: I developed the project using OpenAI API / platform credits and Cursor with Codex-style agentic coding (Codex desktop login was blocked in Pakistan due to phone verification, so the build ran through platform.openai.com + Cursor).
- GPT models: Used for attack planning concepts and LLM-as-judge evaluation patterns in the eval loop.
- Stack: Python, LangGraph, FastAPI, React, PyTorch ecosystem; CLI + REST API; MIT-licensed open source.
Challenges
Designing reliable eval rubrics across languages, keeping demos reproducible offline with a stub target agent, and building a harness that is useful for developers — not just a chat wrapper.
What's next
OWASP ASI mapping UI, connector packs for LangGraph production agents, and Pakistan-focused packs for banking/education chatbots.
Log in or sign up for Devpost to join the conversation.