Tryda

Tryda is a real-time diagnostic and 24/7 accuracy monitoring platform for production AI chatbots and LLM-driven applications.

Comment

# ✦ The Tryda Story: Catching AI Drift Before Your Customers Do

Inspiration

The idea for Tryda came from watching a production customer support chatbot fall apart in real time. In just three conversational turns, a bot built to protect a company's policies was flustered by a mildly demanding user, hallucinated pricing information, and ultimately agreed to a full refund plus three months of free premium access — all to avoid a bad review.

That moment crystallized a problem we saw everywhere: companies were rushing to put LLMs on the frontline of customer support, but a static system prompt and a handful of manual test queries can't anticipate the chaos of real production traffic. Real users don't follow scripts — they push boundaries, demand exceptions, ask trick questions, and apply adversarial pressure. AI agents are fragile under that kind of pressure: they drift, they hallucinate, and they crack.

So we asked: what if a system could continuously stress-test, monitor, and evaluate production chatbots the way a human QA team would — but 24/7/365, with zero manual effort? That question became Tryda: an autonomous diagnostic and accuracy monitoring platform that catches AI drift before it reaches your customers.

How We Built It

Tryda is a modular, high-performance monorepo organized into three layers:

Tryda/
├── frontend/        # Next.js (TypeScript) + TailwindCSS client-side SPA
├── backend/         # Express (TypeScript) API on Google Cloud Run
├── agent/           # Tryda-Agent (TypeScript) automation workspace
└── brand-assets/    # Brand assets, mascot designs, and SVGs

The Reconnaissance & Growth Layer (agent/). Since we wanted Tryda to acquire customers organically without a sales team, we built two autonomous agents that work together as a closed-loop growth funnel:

  • The Prospector Agent runs every two hours and acts as a digital scout. It uses Gemini 2.5 Flash to generate targeted search strings for trending market niches, queries search engines through SerpApi to get around scraping obstacles, and then boots up headless Chromium via Playwright to inspect each candidate site. It performs deep, recursive iframe scanning to detect whether a real, active support widget (Intercom, Zendesk, HubSpot, Crisp, Drift, etc.) is running. Qualified leads are enriched through the Apollo.io API to find the right contact — Head of Support, VP of CX, or CEO — and queued in a Firestore lead_queue.
  • The Auditing & Outreach Agent runs every 25 minutes and acts as the sales closer. It opens a live browser session against a queued prospect's chat widget and uses Gemini 2.5 Flash as an undercover QA tester to run a live, multi-turn adversarial conversation — demanding refunds against policy, threatening bad reviews for freebies, and similar boundary-pushing tactics. The resulting transcript is scored by Tryda's evaluation engine on a 0–100 compliance scale, with specific vulnerabilities and plain-English system-prompt corrections attached. The agent then generates a completely ungated, shareable audit report (tryda.io/audit?id=[leadId]) with the business's name, email, and description already prefilled in the onboarding form, and sends a personalized, non-spammy pitch via the Resend API from our verified domain, aireports@tryda.io, highlighting the exact vulnerability found.

The Evaluation & Analytical Backend (backend/). An Express API running in a containerized Docker environment on Google Cloud Run (us-central1) acts as the central brain. It ingests conversation logs through /api/ingest, runs the monitoring pipeline that turns quality-score drops into actionable recommendations, manages Stripe billing across subscription tiers ($15–$85/mo), and handles auth via the Firebase Admin SDK.

The Client Dashboard (frontend/). Built with Next.js 14, TailwindCSS, and Recharts, the frontend compiles to a fully static SPA (output: 'export') hosted on Firebase Hosting's CDN at tryda.io. It supports Firebase Authentication (email/password and Google Sign-In) and gives customers real-time dashboards tracking conversational health, drift trends, open vulnerabilities, and billing.

Challenges We Faced

The iframe and locator maze. Modern chat widgets live inside heavily nested, cross-origin iframes designed to isolate them from the parent page's styles. Early on, our Playwright crawler would miss widgets entirely or crash when it tried to click buttons intercepted by overlay elements. We fixed this by rewriting the scanner to perform deep, recursive iframe traversal — auditing each frame independently — and adding a retry mechanism that could bypass click interception.

Ephemeral disk and partition limits. In our stateless cron environment, Playwright's default behavior is to re-download the full Chromium binary on every single execution. That quickly filled up the VM's ephemeral /tmp partition and crashed the job. We solved this by caching the browser binaries in a persistent partition at /var/tmp/playwright-browsers, which eliminated the disk bloat and cut execution startup time by roughly 80%.

Silent cron failures. Lead generation runs on scheduled cron jobs on the host, but cron executes in a highly restricted, non-interactive shell. Our scripts initially failed silently with errors like ts-node: command not found and npm: command not found because of environment path mismatches. We tracked down the correct node paths, restored the missing packages directly in the local workspace, and loaded a properly configured crontab into the active process scheduler.

What We Learned

  1. Adversarial AI probing is essential. Guardrails can't be validated statically — dynamic, Gemini-driven adversarial dialogues are the only reliable way to surface flustered behavior, hallucinations, and policy breaches before real users ever encounter them.
  2. Value-first acquisition wins. The most effective way to win over a B2B SaaS customer is to show them their own weakness for free. Generating a detailed, ungated audit report and prefilling the signup form removed nearly all the friction from onboarding.
  3. Symbiotic autonomous agents can drive a business loop end-to-end. With the Prospector continuously surfacing new opportunities and the Auditor qualifying and pitching them, we built a closed-loop system that drives user acquisition and subscription revenue without manual sales effort.
  4. Performance-optimized monorepos pay off. Keeping the frontend, backend, and agent workspaces isolated but shareable in a single repository sped up development significantly while keeping dependency scopes clean.

What's Next for Tryda

  • Broader widget coverage. Extend the Prospector's detection heuristics beyond Intercom, Zendesk, HubSpot, Crisp, and Drift to catch more platforms and fully custom-built bots.
  • A fuller diagnostic suite. Move past a single compliance score toward dedicated checks for hallucination rate, brand-voice/tone consistency, and policy-specific rule violations.
  • One-click remediation. Turn the plain-English system-prompt corrections into a one-click "apply fix" flow so customers can patch a discovered vulnerability directly from the dashboard, instead of editing prompts by hand.
  • Always-on protection, not just discovery. Today the agents focus on prospecting and one-time audits; the next step is continuous, always-on monitoring of a paying customer's live bot, with real-time alerts the moment drift is detected.
  • International expansion. Extend adversarial probing and outreach beyond English-language support widgets to reach a global market of AI-powered support teams.

Today, Tryda is fully live and actively crawling the web — helping business owners protect customer trust, safeguard their bottom line, and run AI assistants that hold up under real pressure.

Built With

Share this project:

Updates