Sentinel

Sentinel predicts Singapore dengue clusters 28 days ahead using Databricks AI, giving NEA officers a 4-week head start to pre-position vector control before outbreaks begin.

Comment

Here is the complete project write-up formatted in Markdown with LaTeX math support, ready to be pasted directly into your Devpost submission page.

Inspiration

Dengue is hyperendemic in Singapore, causing severe public health challenges during epidemic surges—reaching 35,201 cases in 2019 (Singapore's worst outbreak in over a decade) and 32,365 cases in 2022.

Currently, vector-control operations by the National Environment Agency (NEA) rely on a reactive posture: field teams deploy door-to-door inspections and fogging after a clinical cluster is officially confirmed (\(\ge 2\text{--}3\) cases within \(\sim 150\text{--}400\text{ m}\) in a single week). This creates a critical 4–6 week biological detection lag:

\[\text{Rainfall / Temp} \xrightarrow{6\text{--}10\text{d}} \text{Larvae} \xrightarrow{12\text{--}15\text{d}} \text{Adult Biting} \xrightarrow{7\text{--}14\text{d}} \text{Viral Incubation} \xrightarrow{4\text{--}14\text{d}} \text{Clinical Symptoms}\]

By the time hospital reports trigger an address-level cluster, infected adult mosquitoes have already been breeding, biting, and spreading the virus for weeks. Sentinel was inspired by a simple question: What if we could use leading weather indicators and spatial contagion signals to forecast cluster risk 28 days ahead, shifting vector control from reactive crisis response to proactive, targeted prevention?

What it does

DengueRadar is a predictive early-warning system built on the Databricks Lakehouse platform. It models leading environmental drivers to forecast the 28-day forward probability of a dengue cluster emerging across Singapore's planning areas:

\[P(\text{dengue cluster confirmed in area } P \text{ within the next 28 days})\]

Key features include:

  • Ranked Top-5 Priority List: Outputs a weekly prioritized list of the top-5 highest-risk planning areas, enabling NEA headquarters to pre-position finite inspection staff \(2\text{--}4\) weeks before clinical cases appear.
  • Risk Tiering: Buckets predictions into Low, Medium, and High Risk categories.
  • TreeSHAP Explainable Risk Drivers: Displays human-readable drivers for every prediction (e.g., "High Risk: 120mm rainfall in prior 14 days + 150m proximity to active cluster boundary"), guiding officers on what micro-environments to inspect.
  • Confidence Gating: Explicitly flags low-activity areas with low out-of-sample confidence (\(\text{AUC} < 0.50\)) as Low Confidence to prevent ungrounded predictions.
  • 3D Operations Control Panel: Visualizes active clusters, planning area risk zones, and extruded HDB footprints in an interactive dark-mode map interface.

How we built it

DengueRadar was built end-to-end on Databricks utilizing public datasets from data.gov.sg:

[ Ingestion (Bronze) ]           [ Transformation (Silver) ]         [ ML Engine (Gold) ]              [ Serving & UI ]
NEA Weather APIs (5-min)  ──►  Delta Live Tables (DLT)      ──►  XGBoost + MLflow            ──►  Databricks App (FastAPI)
NEA Cluster GeoJSONs           Expect() Quality Gates             Biological Lag Features              Dark 3D Control Panel
URA MP2019 Boundaries          Spatial Point-in-Polygon           Walk-Forward Validation              (deck.gl / MapLibre)
  1. Ingestion (Bronze Layer): Polled live 5-minute real-time weather APIs (rainfall and air temperature) from NEA via Lakeflow Jobs into raw Delta Bronze tables using Lakeflow Auto Loader. Historical weekly NEA Dengue Cluster GeoJSONs and URA Master Plan 2019 boundary layers were ingested alongside.
  2. Data Engineering (Silver Layer): Built Delta Live Tables (DLT) pipelines featuring built-in expect() quality rules to quarantine null storms or sensor outages. DLT executes spatial point-in-polygon joins to map weather stations and cluster polygons onto URA Planning Areas.
  3. Machine Learning & Feature Store (Gold Layer): Engineered biological lag features capturing:
    • Short-term drivers (0–14 days): Cumulative rainfall sweet spots and Extrinsic Incubation Period (EIP) temperature acceleration up to \(\sim 32^\circ\text{C}\).
    • Medium-term drivers (1–4 weeks): Days since last heavy rain, rolling 28-day statistics, and spatial contagion (active clusters in adjacent planning areas).
    • Trained Gradient Boosted Decision Trees (XGBoost) tracked and registered in MLflow under Unity Catalog governance.
  4. Serving & UI: Deployed a Databricks App (FastAPI) that serves precomputed risk vectors and SHAP drivers to a dark 3D deck.gl / MapLibre GL control panel.

Challenges we ran into

  • The Active-Snapshot Label Trap: Raw weekly NEA cluster GeoJSON feeds re-report ongoing active clusters across multiple weeks. Aggregating these raw feeds naively caused severe label leakage and inflated historical annual cluster counts by \(2\text{--}3\times\). We engineered a differential snapshot pipeline in PySpark to track true new cluster onsets vs. ongoing active sites.
  • Class Imbalance & Temporal Data Leakage: In any given week, most planning areas do not have active clusters. Standard random k-fold cross-validation causes severe temporal data leakage across monsoon cycles. We enforced strict rolling walk-forward (expanding window) evaluation across multi-year data to ensure true out-of-sample generalization.
  • Macro Weather vs. Micro Execution: Weather station data is measured at the town level, whereas field officers operate at the precinct level. We resolved this by pairing macro-environmental weather triggers with micro-spatial urban features (low-storey HDB housing density and spatial proximity to active cluster boundaries).

Accomplishments that we're proud of

  • Outperforming Historical Baselines: On rigorous walk-forward out-of-sample testing, our full weather and spatial contagion model achieved an AUC of 0.543, significantly beating the recent-history baseline (AUC 0.392) on area ranking (\(\Delta\text{AUC} = +0.151\)).
  • Resolving the Detection Lag: Successfully demonstrated an operational pipeline that provides a 2–4 week forward lead time ahead of clinical cluster confirmation.
  • Production-Grade Lakehouse Architecture: Implemented a unified Databricks workflow spanning automated streaming ingestion, DLT quality enforcement, MLflow model tracking, and FastAPI container serving under Unity Catalog.

What we learned

  • The Intervention Paradox: In vector control, predicting high risk followed by preventive larviciding that prevents a cluster counts as a statistical "false positive" in label data, but represents a complete operational success in public health.
  • Evaluation Alignment: The importance of evaluating public health ML models using Precision@5 / Hits@5 rather than naive global accuracy, directly reflecting how field teams allocate limited manpower.
  • Data Rigor: How subtle snapshot logging mechanics in public datasets can corrupt ground-truth target labels if not audited carefully.

What's next for Sentinel

  • Micro-Precinct Downscaling: Incorporate high-resolution urban Digital Elevation Models (DEMs), PUB drainage layers, and estate-level construction site registries to narrow planning area risk down to specific HDB sub-zones.
  • Automated Alerting Webhooks: Connect MLflow prediction triggers to automated Telegram and SMS notification pipelines for Town Councils and Grassroots Organizations (GROs).
  • Field Trial Validation: Partner with public health stakeholders to run a pilot study measuring the Outbreak Avoidance Rate (OAR) in top-ranked planning areas during upcoming monsoon surges.

Built With

  • ai
Share this project:

Updates

Submission history