Guardpost

Technology Stack: Backend API & Gateway: Built using FastAPI, Uvicorn, and Pydantic to handle asynchronous request proxying and header metadata extraction (x-risk-tier, x-task-id, x-tenant-id). Detection Pipeline: Python Regex, N-gram claim decomposition algorithms, custom PII scanners, and heuristic prompt injection classification. RAG Engine: Embedded doc search module with term frequency vectorization for contextual retrieval. Frontend Dashboard: Built with Next.js 14, React, Tailwind CSS, and Lucide Icons providing real-time telemetry graphs, live trace detail modals, policy filter chips, and automated stress testing triggers. Database & Storage: SQLite with indexing for sub-millisecond trace reads and persistence. 🚩 Challenges We Faced Minimizing Proxy Latency Overhead: Adding security checks in line with LLM requests risks introducing unacceptable latency. We overcame this by splitting execution into Pre-flight Inline Checks (outbound PII & injection) and Post-flight Verification (groundedness & claim breakdown), optimizing the pre-flight check time down to single-digit milliseconds ($< 5\text{ ms}$).

Catching Numerical & Entity Hallucinations: Standard embedding distance metrics often score text with hallucinated numbers as "high similarity." We solved this by adding an explicit entity extraction pass that penalizes claims containing numbers or dates missing from the underlying RAG context.

Handling Risk Granularity Across Different Verticals: A medical diagnosis tool cannot be evaluated with the same risk tolerance as an internal dev support chatbot. We resolved this by introducing Risk Tiers (0 to 3) and Action Reversibility metadata into the evaluation engine, ensuring context-aware verdicts.

🎓 What We Learned Verification Must Be Context-Aware: Blindly blocking low-confidence responses degrades user experience; attaching caveats (EDIT) or routing to human review (ESCALATE) provides a far better operational compromise. Developer DX is Everything: Requiring engineers to adopt new complex SDKs causes friction. By building GuardPost as an OpenAI-compatible drop-in proxy, integration takes under 60 seconds. Metadata Is Mandatory for AI Telemetry: Raw prompts and completions aren't enough—attaching route IDs, risk tiers, and tenant IDs is essential for root-cause analysis and compliance auditing. 🚀 What's Next for GuardPost 🔹 Streaming Proxy Support: Implementing SSE (Server-Sent Events) token-by-token evaluation and dynamic stream chopping when safety violations occur mid-generation. 🔹 Vector Embedding Integration: Upgrading the groundedness detector with dense semantic vector embeddings (e.g., via sentence-transformers or Gemini Embeddings API) for cross-lingual claim verification. 🔹 Automated Policy Tuning: Self-learning threshold adjustments based on feedback loops from human review escalations.