Inspiration

From the hackathon prompt directly: Databricks brief A2 asks for a 2-week dengue risk view for each Singapore planning area. Dengue is a recurring public-health problem in Singapore, and the people who act on it (inspection teams, town councils, residents) don't read notebooks. We want to hand them a plain answer for each area: do something now, keep an eye on it, or relax.

What it does

DengueRadar is a planned map-and-list app that gives every Singapore planning area one recommended action for the next two weeks:

Act now: an active dengue cluster, and conditions favour spread.
Contain: an active cluster, but conditions are less favourable.
Watch: no cluster yet, but the area looks likely to get one, so inspect before cases appear.
Low: neither signal.

Each area gets two separate signals:

Now: active clusters from NEA's daily clusters feed.
Outlook: where a new cluster is likely in the next 2–6 weeks. We plan to combine residents (Census 2020), local rainfall over the last 8 weeks relative to the island average, and NEA high-Aedes zones.

Each area also gets plain-English reasons (for example, "wetter than the island average over the last 8 weeks"). A national 2-week case forecast sits alongside the map for context. A SQL alert would email when an area newly turns High.

How we built it

Everything is planned as one Databricks Asset Bundle on Databricks Free Edition (serverless only), so a judge can reproduce it with a few commands.

Ingestion: a daily job pulls NEA rainfall and temperature, the dengue clusters GeoJSON, and weekly case counts into a Unity Catalog Volume. A one-off backfill loads weather and MOH case history, planning areas, hawker centres, parks and Census data. Auto Loader reads new files incrementally.
Transformation: a Lakeflow Declarative Pipeline with bronze, silver and gold layers. Stations and clusters are tagged to planning areas through H3 cells, and data-quality expectations guard each layer.
Governance: all tables live in Unity Catalog (the workspace catalog, because Free Edition can't create new ones). The app's service principal gets read-only grants on the tables it uses.
Analytics and AI: a weekly training job compares simple baselines against ridge and LightGBM models across several feature sets. Runs are tracked in MLflow, and the best model is registered in Unity Catalog as champion. A daily scoring job writes the area forecast to a gold table.
App: a Databricks App (Streamlit and pydeck) reads from a serverless SQL warehouse.

Challenges we ran into

These are the risks we've identified in planning and designed around.

No area-level case history is public. Historical cases exist only as national weekly totals. The national model will train on those, and the area layer will only be evaluated (never trained) against archived cluster listings.
Small data, easy to overfit. About ten years of weekly totals can't support many features, so we'll keep the feature set small and compare every model against naive baselines.
Free Edition limits. There's no catalog creation, compute is serverless only and quotas are tight. We'll refresh daily rather than hourly and aggregate the large weather history once.
Data terms. Scraping NEA's weekly counts page and caching OneMap geocodes both raise licence questions. We'll ask the organisers and NEA before relying on them, and keep a fallback.
Hand-set thresholds. Cut-offs such as "flag at 1.2× average outlook" can't be learned from the data we have, so they'll be named constants that we state openly.

Accomplishments that we're proud of

Nothing is built yet, so these are what we're proud of in the plan:

The brief is turned into an action per area rather than a risk score nobody knows how to use.
The design is honest about its evidence. We commit up front to beating a strong naive baseline (last week's count), not only the 4-week average, and to reporting where weather helps and where it doesn't.
The whole system fits on a free workspace and deploys from one bundle.

What we learned

Planning taught us what we need to test, not what the answers are:

Weather may not add much over case history for a national forecast, and we need a rolling back-test across several years to find out.
Population may explain most of where new clusters start, so we'll compare our outlook against a population-only ranking and report the result either way.
A small number of well-chosen signals with clear reasons will probably serve a non-technical user better than a large model.

What's next for DengueRadar

Build the pipeline, train the national model, and run the back-tests above. Then update this write-up with the real numbers.
Resolve the NEA and OneMap data-use questions, and switch to a permitted source if needed.
Ask NEA or MOH about area-level case data, so the outlook can be trained and not just evaluated.
Add alerts for inspection teams and town councils, and test the wording with non-technical users.
Review the hand-set thresholds once there's enough data to tune them.