Semi-Automated Versioned Evals

• 

  chat UI allows product manager or QA to test and copy a weave trace

• 

  product manager makes ticket, a daytona sandbox for claude code is immediately created

• 

  bot makes a PR, comments on the ticket and changes the status of the ticket

• 

• 

  vercel deploys the branch with the latest agent code, product manager or QA can test and view evals

• 

  ui shows eval runs on branch

• 

  eval outputs UI shows tool calls and message bubbles the same as users are used to in the chat

Inspiration

I've felt the pain of tedious prompt iteration, as well as the pain of chaotic QA on an agentic application. This project automates the AI engineering process within a framework of software discipline.

What it does

An AI-engineer bot automatically acts on tickets and makes PRs. It also publishes a UI so product managers can test the branch in a chat playground and view the latest eval results.

How we built it

Claude code runs in a Daytona sandbox, triggered by a Linear ticket. When it finishes, it creates a GitHub PR and the branch is automatically deployed by Vercel. A UI interface wraps Github actions to show evals on that branch. Evals are run with multiple seeds, and cached if inputs have not changed. This helps handle the non-deterministic nature of model outputs, but avoids rerunning every LLM call in every CI job to save time and money.

Challenges we ran into

Long iteration cycle to debug environment issues when claude code in a sandbox

Accomplishments that we're proud of

• e2e integrations working with linear, github, daytona, vercel, claude code
• UI to view eval results that wraps github actions

What we learned

• agent ecosystem in TypeScript might not be mature enough yet (TypeScript support was missing for google adk and litellm, for example)

What's next for Semi-Automated Versioned Evals

• require full eval-driven-development ('TDD') from claude code (it takes more time and the sandbox was timing out)
• use a pre-built Daytona image for faster startup with claude code
• automate not just the AI-engineering step, but product management and data science parts of the development work (looking at production logs and customer feedback, proposing changes that work backwards from business goals).
• try it in a real agent development process and get feedback

Built With

• daytona
• gemini
• helicone
• linear
• mastra
• vercel
• weave