RootChain

RootChain is the Future of On-Call

Comment

Inspiration

I have seen a lot of time wasted during debugging.

The information needed to understand a bug usually already exists somewhere in the development process. It is in merge requests, issues, commit history, code reviews, or deployment records.

The problem is that all of this information is disconnected from the runtime error that engineers see when something fails.

While exploring GitLab Orbit, I realized that the SDLC graph already contains the relationships needed to connect these pieces together.

I wanted to see if I could start from a stack trace and automatically walk backwards through the software development lifecycle to find the change that most likely introduced the problem.

That idea became RootChain.

What it does

When a production issue happens, the hardest part is often figuring out which code change caused it.

Most engineers start with a stack trace and then spend time digging through merge requests, issues, commit history, and team chats to understand what changed and why.

I built RootChain to automate that process.

RootChain watches for GitLab issues that contain stack traces. It extracts the failing files and functions, queries GitLab Orbit, and traces those code locations back to the merge requests that modified them.

It then posts a comment directly on the issue showing:

  • The most likely merge requests involved
  • The developer who made the change
  • The original intent behind the change
  • A confidence score for each candidate
  • Suggested people to involve in the investigation

RootChain also generates a prompt that can be pasted into GitLab Duo Chat to help create a fix.

The goal is simple: reduce the time between "something broke" and "I know where to look".

How we built it

RootChain is built using GitLab Duo Agent Platform and GitLab Orbit.

The workflow looks like this:

  1. A GitLab issue is created with a stack trace.
  2. A Duo Agent workflow starts automatically.
  3. The stack trace is parsed to identify files and functions.
  4. Orbit is queried to find merge requests related to those files.
  5. Additional information such as authors, linked work items, and pipeline status is collected.
  6. A scoring algorithm ranks the most likely root causes.
  7. A report is posted back to the GitLab issue.

I also implemented fallback strategies for situations where Orbit indexing is incomplete or delayed. In those cases, RootChain can verify results using the GitLab REST API.

The project is written primarily in Python and uses async requests to query Orbit efficiently.

Challenges we ran into

The biggest challenge was accurately connecting stack trace frames to the correct merge requests.

There are many edge cases. Files get renamed. Multiple merge requests can touch the same file. Recently merged code may not be indexed yet. Some stack frames come from libraries rather than application code.

I spent a lot of time building fallback strategies and ranking logic so RootChain could provide useful results even when the graph was incomplete.

Another challenge was avoiding hallucinated answers. I wanted every result to be backed by actual GitLab data rather than generated guesses.

Accomplishments that we're proud of

Creating a working GitLab Duo Agent workflow from end to end, since this is my first time with using GitLab :)

What we learned

This project taught me a lot about GitLab Orbit and how much information already exists inside the SDLC graph.

I also learned that debugging tools need to be trustworthy. Engineers are much more likely to use a tool if they can see the evidence behind the recommendation.

That is why I focused on deterministic graph traversal and verification rather than relying entirely on an LLM.

What's next for RootChain

I would like to expand RootChain beyond incident response.

Some ideas I want to explore are:

Automatic rollback recommendations CI/CD failure investigation Security incident analysis Better support for monorepos Cross-service dependency tracking Automatic routing of incidents to the right teams

The long-term goal is to make GitLab's SDLC graph useful during production incidents, not just during development.

Built With

  • duo
  • flows
  • gitlab
  • orbit
Share this project:

Updates