Testah — Before AI Agents Spend Money, Test Them.
Inspiration
AI agents are evolving from simple assistants into autonomous systems capable of shopping, interacting with merchants, and making financial transactions.
But this raises an important question: How can we trust AI agents with money if we haven't tested their financial decision-making?
That question inspired us to build Testah, a testing playground where developers can evaluate their AI agents before deploying them in real-world financial environments.
What It Does
Testah is an AI-powered testing platform for agentic commerce.
Developers connect their own AI agents, and Testah uses Gemini to generate personalized financial challenges within a controlled testing environment.
Agents navigate merchant interactions, handle payment scenarios through PayPal Sandbox, and encounter challenges involving misleading listings, unsafe transactions, and unexpected situations.
Testah monitors their decisions and generates personalized, evidence-based reports highlighting performance, safety, strengths, and areas for improvement.
How We Built It
We developed Testah using:
- React and TypeScript for the interactive dashboard.
- FastAPI and Python for backend services and agent orchestration.
- Google Gemini to dynamically generate personalized testing scenarios.
- PayPal Sandbox APIs to test payment workflows without transferring real money.
- AG Grid to visualize execution evidence and assessment results.
- APIMatic to generate developer-facing API documentation.
Our architecture separates challenge generation, agent execution, payment processing, and evaluation to make assessments structured and reproducible.
Challenges We Faced
One major challenge was designing a system that could test external AI agents without knowing their internal architecture.
We addressed this by introducing a standardized HTTP-based agent communication protocol.
Another challenge was balancing AI-generated scenarios with reliable evaluation. We combined Gemini-powered challenge generation with deterministic scoring and critical safety checks.
Integrating financial workflows also required careful handling of authorization, transaction evidence, and secure Sandbox execution.
What We Learned
Building Testah helped us understand agentic commerce, external agent orchestration, secure payment APIs, and the importance of evidence-based AI evaluation.
Most importantly, we learned that AI autonomy must be accompanied by accountability and safety.
What's Next?
Our vision is to evolve Testah into a developer testing infrastructure for agentic commerce.
Future improvements include broader financial scenarios, automated regression testing, isolated developer workspaces, and deeper integrations with agent frameworks.
Before AI agents spend money, test them.
Built With
- ag-grid
- ai
- alembic
- apimatic
- fastapi
- github
- google-gemini
- httpx
- javascript
- paypal
- paypal-sandbox
- postgresql
- pydantic
- python
- react
- render
- rest-api
- sqlalchemy
- typescript
- vite
