Critical Path Guardian: Real-time Incident Insights

Imagine looking at main business metrics during the incidents and understanding right there what is wrong or what has changed. The project provides insight into the complex systems during incidents.

Comment

Inspiration

I think all engineers experienced frustration from not finding answers quick during the incident. Sometimes the most simple code changes combined together can create really complex and obscure problems.

There is an easy way to build a dashboard with all releases across the company and there are AI tools doing code reviews on codebases. But during the incident engineers need a quick overview of latest events and a glimpse of what the event/change was without spending too much time on it. This project aims to fill this gap by enriching deployment/event information with an actual code change summary.

What it does

(Critical Path Guardian) CPG is intended for the following:

  • mark easily the service as business critical by adding Gitlab CI/CD Component to the pipeline
  • annotate the release with information about the actual code change analyzed by AI with a quick assessment of risks

How we built it

It was built using Gitlab Ci/CD Component, additional Agent service on top of cloud native landscape and Kubernetes. More information is here

Google cloud platform was used extensively for provisioning GKE (Kubernetes) and GitLab Runner software. Besides that Vertex AI from Google was used for analysis.

Accomplishments that we're proud of

It's working and provides actually quite nice insights into the code change!

What we learned

The Grafana annotations are quite tricky and buggy. The whole mechanism of storing this data in metrics is unusual but I wanted to have a maximal compatibility with existing tools.

What's next for Critical Path Guardian: Real-time Incident Insights

Battle test it in production and hopefully see some interest from the community.

Built With

Share this project:

Updates