Lahore Air Quality Data Warehouse & Decision System

Inspiration Lahore frequently experiences hazardous winter smog episodes where PM2.5 concentrations exceed WHO safety thresholds by over 10x. Citizens, schools, and health officials lack reliable, automated, and early-warning insights to make timely protective decisions. Our team was inspired to bridge this gap by building a rigorous, verified data warehouse and decision-support system that enforces Punjab EPA school safety rules with transparent uncertainty tracking and grounded AI assistance.

What it does Automated MLOps Pipeline: Runs hourly Prefect orchestrations fetching incremental sensor data from OpenAQ v3 and Open-Meteo weather APIs into immutable raw storage.

Data Warehouse & Governance: Enforces strict Pandera schemas, 18-of-24-hour daily completeness rules, and SHA-256 record hashing, logged via an automated Data Quality Agent.

Leak-Free Predictive Modeling: Implements a rigorous Histogram GradientBoosting Regressor (HistGradientBoosting_ResidualDelta) forecasting next-day PM2.5 residual shifts with 80% pinball loss uncertainty bounds.

Cited RAG Policy Assistant: Features an Insight Agent leveraging primary database keys (record_id) and official World Health Organization (WHO) and Punjab EPA guidelines to answer user queries with 100% grounded citations and zero hallucinations.

Interactive Multilingual Dashboard: Deployed live on Hugging Face Spaces with freshness badges, model evaluation metrics, and an Urdu translation toggle for local accessibility.

How we built it Data Layer & Warehouse: Python, Pandas, DuckDB, Parquet, OpenAQ API, Open-Meteo API.

Orchestration & MLOps: Prefect, Pandera schema validation, GitHub Actions (v1.0-final release).

Machine Learning & AI: Scikit-Learn (HistGradientBoosting), BM25 / RAG vector embeddings, custom uncertainty interval calculations.

Frontend & Deployment: Streamlit, Hugging Face Spaces.

Challenges we ran into Handling upstream API irregularities and missing sensor nodes from OpenAQ without breaking downstream analytics. We solved this by implementing an automated Data Quality Agent with non-fatal logging and strict 18-of-24-hour daily completeness validation.

Preventing data leakage in temporal machine learning features, which we resolved by designing a strict residual-delta modeling structure.

Ensuring zero hallucinations in the RAG policy assistant, achieved by enforcing primary key database record tracing (record_id) for every generated citation.

Accomplishments that we're proud of Successfully orchestrating an automated, end-to-end data pipeline from raw JSON ingestion to a fully deployed live Streamlit dashboard.

Achieving a robust Test MAE of 12.594 µg/m³ with verified uncertainty bounds and strict adherence to the official 10-point competition checklist.

Maintaining a fully transparent repository structure tagged at v1.0-final with comprehensive audit logs (prefect_runs.csv, manifest.csv, and AI_Use_Log.md).

What we learned The critical importance of rigorous data governance, schema enforcement, and immutable raw data layers when building trustworthy civic AI applications.

How proper MLOps tooling (Prefect) and automated logging streamline reproducibility and eliminate deployment bottlenecks under tight deadlines.

What's next for Lahore Air Quality Data Warehouse & Decision System Integrating satellite aerosol optical depth (AOD) data for spatial interpolation across wider Punjab regions.

Expanding sensor node coverage and partnering with local educational networks to pilot automated Punjab EPA school safety alerts in real time.

Built With

• duckdb
• openaq
• openmateo
• pandas
• prefect
• python
• scikit-learn
• streamlit