Inspiration
One founder. A team of AI employees. But who's checking their decisions?
Starting a company used to require a team of developers, marketers, customer-support representatives, and operations staff. Today, AI is changing that.
A single founder with a laptop can build software, serve customers, automate workflows, and launch products faster than ever before.
But as AI evolves from answering questions to taking actions, a new problem emerges.
What happens when an AI employee makes a financially costly mistake?
An AI refund assistant might refund the same customer three times. A sales agent might offer an unauthorized discount. A fraud-detection agent might block legitimate payments.
These mistakes aren't always obvious from the conversation. An AI agent can sound perfectly helpful while violating a company's business rules.
That realization inspired LossLab — a crash-testing platform for AI agents before businesses trust them with money.
Our goal is simple: Make AI-powered businesses not just faster to build, but safer to operate.
What it does
LossLab is an AI-agent risk-testing platform that detects business-rule violations, models their potential financial impact, and tests safeguards before deployment.
It follows a four-step process:
1. Simulate agent behaviour
Choose an AI-agent profile, such as an airline refund assistant, SaaS discount agent, or fraud-operations assistant. LossLab runs reproducible scenarios representing financially consequential decisions.
2. Detect policy violations
A deterministic policy engine evaluates structured agent actions against explicit business rules, including duplicate transactions, approval requirements, monetary limits, and escalation constraints.
Instead of producing a vague risk score, LossLab identifies the rule, the offending action, and the supporting evidence.
3. Quantify financial exposure
LossLab calculates modeled financial exposure using transparent assumptions about incident frequency and economic impact.
For example, our fictional airline's refund agent attempts three $130 refunds for the same booking. Two are duplicates, representing $260 in potential excess refunds per incident.
Assuming 24 such excess refund actions per month, the modeled exposure is $3,120 per month.
Across all 14 failing airline scenarios, the synthetic baseline totals $12,437 in modeled monthly exposure.
4. Apply safeguards and retest
This is LossLab's key feature.
A developer can enable protections such as transaction idempotency, trusted-source boundaries, and currency validation.
LossLab reevaluates the same action traces with those protections enabled, showing which failures are intercepted and which risks remain.
In our airline simulation, the selected safeguard suite reduces modeled monthly exposure from $12,437 to $2,741 — a 77.96% modeled reduction.
These are synthetic risk estimates, not measured customer losses or proven real-world savings.
How we built it
LossLab was developed as a full-stack web application with a focus on reproducibility, explainability, and commercial scalability.
Technology stack:
- Next.js and React for the web application
- TypeScript for the simulation and risk engines
- Tailwind CSS for the interface
- Zod for structured data validation
- Vitest for unit testing
- Playwright for end-to-end browser testing
- Vercel for public deployment
The architecture separates four core systems:
Policy Evaluation Engine: Processes structured agent-action traces and detects violations through deterministic rules.
Financial Risk Model: Calculates scenario-level and aggregated economic exposure using integer-cent arithmetic and explicit assumptions.
Remediation Engine: Replays identical test scenarios with different safeguards and recalculates exposure from evaluated outcomes.
Interactive Dashboard: Presents evidence, financial impact, safeguard controls, detailed reports, and developer integration documentation.
A major architectural decision was to keep evaluation logic independent of the interface. All financial figures and visualizations derive from the evaluated results rather than separately maintained display values.
The current MVP uses synthetic fixtures and does not require external AI APIs or customer credentials. This makes the demonstration repeatable and avoids making unsupported claims about testing production AI agents.
Challenges we ran into
Making risk calculations trustworthy
It was important to distinguish the cost of one simulated incident from projected monthly exposure. We built the financial calculations around explicit assumptions and ensured that baseline, prevented, and residual exposure reconcile.
Testing real safeguards instead of displaying fake improvements
Simply changing a risk number after clicking a button would not prove anything. We implemented a remediation process that reevaluates the same structured traces when safeguards are enabled.
Balancing technical complexity with usability
Our initial interface was heavily inspired by complex financial dashboards. During development, we realized that the most important innovation wasn't the visualization itself — it was helping users understand exactly what went wrong.
We redesigned the product around clear action timelines, understandable explanations, and a before-and-after safeguard comparison.
Browser compatibility and deployment reliability
We encountered styling differences between development and production configurations, Content Security Policy restrictions, and browser-test interference. We resolved the relevant build and testing issues, improved test isolation, and verified the deployed workflow.
Accomplishments that we're proud of
- Built a functional deterministic policy-evaluation engine.
- Implemented reproducible agent-action simulations across three fictional business profiles.
- Connected policy findings to financial-exposure calculations.
- Built a working safeguard-and-retest experience.
- Created transparent reports that reconcile baseline, prevented, and remaining exposure.
- Developed a clean, responsive interface that makes complex risk analysis understandable.
- Deployed a publicly accessible MVP.
- Passed the three end-to-end browser tests against the live deployment.
Most importantly, LossLab demonstrates the entire journey from identifying a financially dangerous agent action to testing whether a safeguard prevents it.
What we learned
Building LossLab changed how we think about AI safety.
It's not enough to ask whether an AI agent generates good responses. We also need to ask whether its actions are authorized, repeatable, and economically safe.
We learned the importance of deterministic testing, separating financial assumptions from observed behaviour, and designing technical infrastructure around a problem users can understand.
We also learned that making a sophisticated product simple to use can be just as challenging as building the underlying technology.
What's next for LossLab
LossLab is currently a functional MVP using synthetic simulations. Our next goal is to validate its usefulness with real AI engineering teams and develop integrations for customer-owned agents.
Our roadmap includes:
Phase 1 — Developer integrations
Allow companies to submit their own structured agent traces and business policies through an SDK or API.
Phase 2 — Continuous testing
Integrate risk regression tests into development pipelines so engineering teams can detect policy failures before releasing agent updates.
Phase 3 — Scalable AI risk infrastructure
Support multiple agent types, custom policy libraries, historical reports, team workflows, and organization-level monitoring.
Business model
LossLab would initially target AI startups, ecommerce automation teams, and businesses deploying customer-support or transaction-handling agents.
Potential revenue would come from subscription plans, usage-based testing, and enterprise integrations. Pricing and customer demand remain hypotheses to validate.
The long-term vision is to make financial-policy testing a standard part of building AI-powered businesses.
AI gives founders leverage. LossLab gives them control.
We're building the crash-test layer for the AI workforce.
Built With
- next.js
- playwright
- react
- tailwindcss
- typescript
- vercel
- witest
- zod
Log in or sign up for Devpost to join the conversation.