Braindiff

Inspiration

Agents often look smarter after they get more context, but more context isn’t the same as learning. We wanted a way to prove an AI coding agent actually improves from a lesson it learned earlier, using a test a judge could trust.

What it does

Braindiff runs two versions of the same coding agent on the same held-out bug. One has no memory. The other gets one saved lesson, such as “trace the bad value to its source before patching.” Each patch is graded with an LLM judge that labels it real_fix or hid_error. Arize traces every run, so we can compare pass rate, real-fix rate, latency, and cost per correct fix.

How we built it

Chose a bug from an open source repo and took the comment from that bug fix as the context. Use that context to run a similar bug fix

Challenges

Integrating Surreal

Accomplishments that we're proud of

Solving a real and frequent problem that holds up speed of development

What we learned

Built With

• arize
• claude
• cursor
• daytona
• openai
• surrealdb