Inspiration

India has 14 of the world's 20 most polluted cities, yet AQI numbers alone tell us nothing about whether a city can actually handle a pollution crisis. A city with AQI 150 and 5 hospital beds per lakh is in a completely different situation than one with AQI 150 and 50 beds. That gap — between pollution burden and health readiness — is what AirGuard measures.

What we built

AirGuard is an end-to-end intelligence system built entirely on Databricks. We fused CPCB historical AQI data with NHP health infrastructure records across 26 major Indian cities, running a full Medallion Architecture pipeline (Bronze → Silver → Gold) on Delta Lake with Unity Catalog governance.

The core output is a City Preparedness Score — a 0-to-1 index that combines pollution impact (AQI + PM2.5) with health infrastructure strength (hospitals, PHCs, CHCs). Cities are ranked and tiered: Critically Vulnerable, Moderate Risk, or Well Prepared.

On top of scoring, we trained an XGBoost model (MAE: 14.16 AQI points) to forecast city-level AQI 3 days ahead, tracked with MLflow autologging. Mosaic AI (Llama 3.3 70B) generates a specific, actionable policy recommendation for each city — referencing real Indian government schemes like NHM, PM-ABHIM, and NGT orders.

The full pipeline is orchestrated via Databricks Workflows and the results are surfaced through a Gradio dashboard deployed on Databricks Apps.

What we learned

Serverless compute on Databricks has specific constraints around MLflow experiment paths and Spark config access — we had to adapt our tracking approach accordingly. City-level health data and AQI data use completely different geographic identifiers — significant effort went into standardising city-to-district name mappings across government datasets. Mosaic AI Foundation Models integrate cleanly with mlflow.deployments — no separate API key or endpoint setup needed within Databricks.

Challenges

CPCB data has heavy null coverage in early years (2015-2016) — required city-level mean imputation and AQI recomputation from PM2.5. Health infrastructure data is at facility level (200k rows) — needed aggregation to district level before joining with city AQI data. 7 major cities (Delhi, Mumbai, Kolkata etc.) had name mismatches between the two datasets — resolved with manual mapping and national average fallback.

Built With

  • apache-spark-(pyspark)
  • cpcb-aqi-dataset
  • databricks
  • databricks-apps
  • databricks-volumes
  • databricks-workflows
  • delta-lake
  • folium
  • gradio
  • mlflow
  • mosaic-ai-(meta-llama-3.3-70b-instruct)
  • nhp-health-infrastructure-dataset
  • pandas
  • plotly
  • python
  • unity-catalog
  • xgboost
Share this project:

Updates