Retrace

Retrace turns CI failures into clear, AI-powered diagnoses—so developers can fix bugs faster.

Comment

Inspiration

CI failures are often easy to reproduce but surprisingly time-consuming to understand. A developer may know that a test failed, but the CI log can contain hundreds of lines of output, making it difficult to quickly determine whether the failure is caused by a real regression, a race condition, a hard-coded wait, or an external dependency.

We wanted to build a tool that turns a raw CI failure log into an actionable explanation instead of making developers manually dig through the logs.

What it does

Retrace analyzes CI failure logs and classifies the likely cause of the failure.

A developer can paste or upload a failure log, and Retrace identifies relevant signals in the log and returns:

  • A failure classification
  • A confidence score
  • An explanation of why the failure was classified that way
  • The relevant log evidence behind the classification

Retrace can also integrate directly with GitHub Actions. When a test fails in a pull request, the workflow captures the failure log, sends it to Retrace, and posts the resulting classification back to the pull request.

This means developers can see the likely cause of a failure without leaving their normal GitHub workflow.

How we built it

We built Retrace as a web-based failure analysis tool with a backend classification API and a GitHub Actions integration.

The workflow runs the project's tests, captures the output, and sends the failure information to the Retrace /classify endpoint. The returned classification is then surfaced in the GitHub pull request as a comment.

We also created a simple interface where users can paste failure logs or upload .log/.txt files and immediately receive a classification.

Challenges we ran into

One of the biggest challenges was making the GitHub Actions workflow reliably handle failing tests while still allowing the failure log to be analyzed.

Normally, a failed test causes the workflow to stop immediately. We needed to capture the output, send it to Retrace for analysis, post the result to the pull request, and only then mark the workflow as failed.

We also had to work through CI configuration issues, including dependency installation and GitHub Actions workflow placement, before getting the complete end-to-end flow working.

Accomplishments that we're proud of

We are proud that Retrace works as an end-to-end system rather than only as a standalone classifier.

A failure can go from:

Test failure → CI log → Retrace analysis → classification → GitHub PR comment

We successfully demonstrated this flow with a deterministic failing checkout test where the expected total was 1260 but the application returned 1400. Retrace classified it as a likely real regression with 95% confidence, explaining that the deterministic mismatch pointed toward an application-logic change rather than timing or external factors.

What we learned

We learned that building an AI-powered developer tool involves much more than the AI classification itself. The surrounding engineering workflow is equally important.

Working with GitHub Actions taught us how CI workflows capture failures, manage exit codes, pass logs between steps, and integrate results back into pull requests.

We also learned how important it is to provide evidence alongside an AI classification. A confidence score and explanation make the result much more useful than simply returning a label.

What's next for Retrace

We want Retrace to become a more comprehensive CI debugging assistant.

Future improvements could include:

  • More failure categories and stronger classification accuracy
  • Deeper integration with GitHub Actions and other CI platforms
  • Historical analysis of recurring failures
  • Automatic comparison of failures across commits
  • More detailed suggested fixes
  • Support for larger and more complex CI logs
  • Tracking failure patterns across projects and repositories

Our goal is to make CI failures easier to understand and reduce the time developers spend debugging repetitive or noisy test failures.

Built With

Share this project:

Updates

Submission history