Talk to Test

Turn retail conversations into AI Agent tests cases

Comment

Inspiration

Retail staff interact with countless customers every day, but capturing these conversations for improving AI assistants is tedious and inconsistent. We wanted to find a way to automatically turn real-world interactions into actionable insights, specifically test cases for evaluating AI agents. Our goal was to close the loop between human-AI interaction and AI training/testing, making AI assistants smarter faster.

What it does

Talk to Test equips retail staff with OMI AI wearable devices that capture conversations in real time. After the conversation, the audio is transcribed and automatically converted into structured test cases for AI agents. These test cases can then be used to evaluate the accuracy, helpfulness, and contextual understanding of AI assistants, enabling rapid improvements without manual effort.

How we built it

  1. OMI AI Wearables: Captured live audio from retail staff. Generate an app on Omi Marketplace to have external integration
  2. Webhook Integration: Once transcripts were processed, they were sent to our backend in Supabase via Vercel
  3. OpenAI Conversion: Open AI's API turn unstructured conversations into structured test cases.
  4. Storage & Analysis: Test cases were stored in a database and made accessible for AI evaluation dashboards.
  5. [FUTURE] Automation: The system fully automates the loop from conversation → test case → AI evaluation.

Challenges we ran into

  1. Integrating OMI AI’s webhook reliably to capture and process transcripts in real time.
  2. Automating the flow from OpenAI API responses to Supabase, ensuring test cases are saved and structured correctly.

Accomplishments that we're proud of

  1. Demonstrated the potential for in-person conversations AI agent performance.
  2. Built a scalable workflow that can support multiple staff members and locations.

What we learned

The importance of scoping early: we initially aimed to not only generate test cases but also run them on existing Voice AI agents. Narrowing the focus midway helped us deliver a working prototype faster.

What's next for Talk to Test

  1. Convert the generated test cases into executable tests and run them on Voice AI agents.
  2. Deploy and validate the system in real retail environments. This is particularly valuable for our Voice AI startup, where authentic test cases have been scarce - this approach can quickly provide practical, real-world data for improving customer support automation.

Built With

  • omi
  • openai
  • v0
Share this project:

Updates