PurpleOps

PurpleOps is an AI Security Copilot performing red‑team prompt attacks and deep file risk scans on LLMs and ML models to help teams detect and remediate vulnerabilities before deployment.

Comment 1

Inspiration

The idea for PurpleOps was born from a frustration we constantly faced: while cybersecurity tools and AI models have advanced significantly, AI red teaming and security auditing remained disconnected, manual, and difficult to integrate into workflows. We wanted something different — a modern AI copilot built for security teams, not just researchers. Inspired by the legends of purple teams in cybersecurity — where red (attack) and blue (defense) combine — PurpleOps was designed to be the assistant that merges the offensive creativity of a red teamer with the precision of an AI defender. We envisioned a world where AI could not only simulate attacks but proactively suggest vulnerabilities, assist in CTFs, and guide forensic investigations, all from a clean, intuitive interface. PurpleOps isn't just a tool. It's your tactical AI teammate.

What it does

  • LLM Red Teaming: Users can input natural language prompts to simulate jailbreak attacks, adversarial tests, and evaluate model robustness using dynamic red teaming techniques.
  • Model Risk Analyzer: Upload AI model files (e.g., safetensors, ONNX) and let PurpleOps scan for embedded risks, security flaws, and framework vulnerabilities.
  • Context-Aware Chat: A conversational AI security assistant ("PurpleOps") is integrated directly into the app, offering users guidance, risk assessments, and attack ideas in real-time.
  • Seamless File Uploads: Drag and drop model files directly into the conversation to trigger automated security scans.
  • Stateful Session Management: Previous chats and security sessions are stored and easily retrievable, so users can pick up right where they left off.

How we built it

  • Frontend: Built using Streamlit with custom HTML/CSS tweaks, creating a fast, reactive, desktop-like experience with minimal deployment complexity.
  • AI Integration: Leveraged Anthropic’s Claude 3 Opus models for red teaming, risk evaluation, and natural conversation — mastering prompt engineering and system message conditioning to control AI behavior precisely.
  • File Scanning Engine: Developed a modular "Model Context Protocol (MCP)" server to handle file analysis, framework fingerprinting, and automated risk tagging.
  • Session Management: Architected persistent chat sessions with Streamlit state and smart session handlers to allow multiple security analyses in parallel.
  • Upload Handling: Implemented secure drag-and-drop file uploads with custom error handling to maintain a smooth experience even in high-velocity interactions.

Challenges we ran into

  • Dynamic Action Triggering: Teaching Claude to understand when to automatically initiate file scans, red teaming, or simply respond conversationally required nuanced system prompts and careful state detection.
  • Upload Conflicts: Managing file uploads alongside chat interactions in Streamlit without crashing the session state needed intricate flow control.
  • Prompt Injection Safety: Testing jailbreak prompts without accidentally crashing or hallucinating Claude required layered input validation and fallback protections.
  • Real-Time Feedback Loop: Designing a responsive "assistant" that feels collaborative (without overwhelming users) demanded multiple UI/UX iterations.

Accomplishments that we're proud of

  • Created a full AI red teaming platform that's operational in a single browser tab — no heavy dependencies, no external orchestrators required.
  • Merged conversational AI + file security workflows into a single seamless interface.
  • Built an AI copilot (PurpleOps) that feels natural, trustworthy, and knowledgeable, yet keeps users firmly in control.
  • Designed a modular architecture where new models, attack vectors, and scanning techniques can be plugged in easily.

What we learned

  • Prompt Engineering is UX Design: Writing system prompts for an AI copilot is like writing the entire user experience. It's delicate, strategic work that affects everything from tone to security.
  • Security Tools Need to Feel Effortless: If a security tool requires users to "think" about the interface too much, it already failed. Every upload, scan, or chat needed to be frictionless.
  • Error Handling Should be Invisible: Problems like missing uploads, invalid models, or API hiccups shouldn't crash the experience. Smooth error recovery was essential for user trust.

What's next for Purple OPS

  • Multi-Model Red Teaming: Extending beyond Claude to include GPT-4, Gemini, and open-weight models for comparative red team evaluations.
  • Advanced Risk Scoring: Integrating OWASP LLM Top 10 and model-specific risk profiles for more granular vulnerability detection.
  • Memory-Augmented Agents: Giving PurpleOpsl a memory, allowing it to learn user preferences over time and become even more strategic.
  • Expanded Upload Support: Adding support for full HuggingFace safetensors metadata parsing, ONNX analysis, and even Dockerized AI apps.
  • Mobile Interface: Building a companion mobile experience for quick CTF team discussions and model audits on-the-go.
  • Collaborative Attack Planning: Enabling multiple users to work together on red team plans, adversarial testing campaigns, and AI system audits.

Built With

Share this project:

Updates