Datum

Inspiration

Buying an HDB resale flat has a quiet trap in it. You agree a price with the seller and commit an option fee before HDB tells you what the flat is actually valued at — and if the valuation comes in below the price, the gap is payable only in cash, because no loan and no CPF will touch it. Roughly 24,800 households go through that every year. Sellers arrive with agents and transaction analytics; buyers arrive with a downloaded spreadsheet. We wanted to close that gap, and to do it at the moment the market is hardest to read: the Resale Price Index has just fallen two straight quarters for the first time since 2019, and the 15-month wait-out period was lifted in July 2026.

What it does

Datum rebuilds a constant-quality price index for every town and flat type directly from raw resale transactions, then forecasts it six months ahead with prediction intervals. The distinction is the whole point. A raw median moves whenever the mix of flats sold changes — older, smaller, lower-floor units in one month, newer ones the next — which looks like a market move but isn't. Datum strips that out first, then asks where prices are genuinely heading. An affordability layer sits on top, tracking town-level prices against median household income, so the question becomes not only whether prices are rising but whether a median household can still buy there.

How we built it

The design separates two questions that most projects merge into one. A hedonic model removes flat-mix effects to produce a constant-quality index — the same approach HDB uses for its official RPI, rebuilt from the raw data — and a second model forecasts that index forward. On Databricks, the pipeline ingests data.gov.sg through Lakeflow into Delta, cleans and types the transactions with quality checks, materialises the index as a gold table per town and flat type, forecasts it with a gradient-boosted model tracked in MLflow, and surfaces the result through an AI/BI dashboard and a Genie space, with Unity Catalog for lineage. Before writing the pitch, we profiled the source through the API: 241,710 transactions spanning January 2017 to September 2026 across twelve fields.

Challenges we ran into

Two surfaced before we wrote a line of pipeline code. The first was leakage. Reviewing prior HDB price-prediction projects, we found that most split their data randomly, which places future transactions in the training set and makes the reported accuracy meaningless for anything forward-looking; several also used price per square metre as a predictor, which is simply the target divided by floor area. We now keep a published forbidden-feature list and validate with rolling origins only. The second was sparsity. Segment sizes vary roughly sixtyfold — Tampines 4-room has 6,288 transactions since 2017, Bukit Timah 3-room has 107 — so monthly forecasting is comfortable at one end of that range and pure noise at the other. Thin segments pool up to region rather than pretending to a precision the data cannot support.

Accomplishments that we're proud of

Profiling the data before pitching rather than after. The sparsity finding reshaped the design and the leakage review reshaped our validation strategy, both while they were still cheap to act on. We also resisted the obvious version of this project. A flat-price estimator is the default HDB student build, and it answers a question the market has already answered by the time you ask it. Forecasting the index forward is harder, far less common, and actually useful to someone deciding whether to commit this month or wait.

What we learned

That the interesting problems in Singapore housing data are structural rather than algorithmic. Lease decay isn't linear: it steepens once remaining lease drops below 60 years, because that is where CPF and loan rules start shrinking the buyer pool — so the kink is a financing artefact, not a physical one, and a linear lease term would miss it entirely. HDB's own index is mix-adjusted for precisely the reason our raw medians misbehaved. And policy effects are already embedded in the price history, which means a lag-based model inherits them without ever anticipating them — a limitation worth measuring rather than hiding.

What's next for Datum

The two-week sprint: ingestion and cleaned Delta tables, the constant-quality index, forecasts benchmarked against three baselines on rolling-origin backtests, and the dashboard. After that, the affordability layer and Unity Catalog governance. The extension we most want to reach is treating the model's own error as a signal — when prices run persistently above forecast after a known policy date, that gap becomes a measurable read on what the policy actually did, which is something a quarterly national index cannot offer.

Built With

• python