MLH Hackathon - Quest: Incident Response

Inspiration

I’ve seen that in many systems, things break but nobody notices immediately. Logs are messy, alerts are missing, and debugging takes too long.

I wanted to build something close to what real DevOps/SRE teams use — where you can actually see what’s happening and get alerted when something goes wrong.

What it does

It’s a simple backend app with monitoring and alerting added on top.

Exposes metrics using Prometheus Shows dashboards in Grafana Sends alerts using Alertmanager (Discord webhook) Uses structured logs instead of print statements

Basically, if the app goes down or errors increase, you’ll know

How we built it

Started with a basic Python backend and PostgreSQL.

Then added things step by step:

Logging → structured JSON logs Metrics → /metrics endpoint Prometheus → scraping metrics Alertmanager → sending alerts Grafana → dashboards

Everything runs using Podman containers.

Challenges we ran into

Getting Prometheus and Alertmanager configs right Making sure alerts actually trigger (and not too often) Container networking issues Figuring out what metrics actually matter

Accomplishments that we're proud of

End-to-end setup (logs → metrics → alerts → dashboards) Alerts actually working on failure Clean and structured logs Feels like a real production setup

What we learned

Monitoring is more than just logs Alerts need tuning, otherwise they become noise Metrics design is important Small setups can still feel like real systems

What's next for MLH Hackathon - Quest: Incident Response

Grafana: http://165.22.212.222:3000/ --> test/test123 alertmagerr: http://165.22.212.222:9093 prometheu: http://165.22.212.222:9090 appliaction: http://165.22.212.222:5000

api endpoits: curl -s http://165.22.212.222:5000/health --> Health check

curl -s http://165.22.212.222:5000/product ---> List product

curl -s http://165.22.212.222:5000/product/id --> List product by id

curl -X POST http://165.22.212.222:5000/products -H "Content-Type: application/json" -d '{"name":"Asus laptop","category":"Electronics","price":50000,"stock":10}' --> Add new product to backend

curl -s http://165.22.212.222:5000/metrics --> expose metrics for monitoring

curl -s http://65.22.212.222:5000/error --> For generating High Error request alert (return 500 status)

for i in {1..100}; do curl -s http://65.22.212.2225000/error
done wait