C3

Closing India's Medical Deserts with Data

Comment

Inspiration

Across rural and tier-2 India, NGO planners face the same blocker: a list of "available" hospitals where the data has lied to them. A facility says it has an ICU and there's no ventilator on site. A district shows "covered" because one row is overcounted. A pin on the map exists because someone uploaded a directory five years ago. They cannot tell "no care here" apart from "no data here" — and the difference is the difference between sending an ambulance and sending a clipboard.

What it does

  • H3 care-gap map — India bucketed into hexagons; color encodes score (red→green) and alpha encodes confidence (faded = data-poor). One glance separates a care gap from a data-deficient region.
  • Action Center drill-down — pick any cell or region; see every facility that contributed to its score, with the underlying citations from the source text and an evidence-strength badge (strong / partial / suspicious / none).
  • Genie inline Q&A — non-technical planners ask "Where are the highest-risk maternity gaps in Bihar with high evidence confidence?" and get an answer + the SQL Genie wrote, against the same gold layer the map reads.
  • Scenarios + overrides — save a filtered view, log an override note when a planner has ground-truth that the data lacks, persist both to Lakebase so the next session and the next planner see the same context.

  • Performance page — district-level risk stratification and a real time-series of how scores have moved across ETL runs (no synthetic projections).

    How we built it

  • Medallion ETL on Unity Catalog: Bronze (raw Delta) → Silver (geocoding via DBR's built-in h3_longlatash3, capability claim extraction with citations, NFHS-5 demand join) → Gold (pipelines/gold/trust_weighted_score.py for h3_care_score + state/district rollups; supplemental pipelines/gold/medical_desert.py for demand-aware scoring).

  • Confidence as a first-class signal — confidence ∈ [0, 1] blends facility coverage, average claim weight, and source-URL coverage; cells below DATA_DEFICIENT_THRESHOLD are coloured grey rather than red, so weak evidence never masquerades as "proven absent care".

  • Genie binding declared on the App resource (config/resources/app.yml) — the App calls Genie with CAN_RUN, the space content is provisioned by scripts/setup_genie.sh.

  • Lakebase (managed Postgres) for write-side state — scenarios, overrides, gap categorisations, shareable bookmarks via app/services/lakebase.py.

  • Streamlit + pydeck front end with a single brand-driven CSS layer in app/services/brand.py.

  • Databricks Asset Bundle for repeatable deploys (databricks.yml, deploy.sh) — one command provisions Job, App, Genie binding, UC schemas + tags, and triggers the ETL.

  • Metadata-driven ingestion — adding a new source is editing one YAML in config/ingestion/, not writing a new pipeline.

    Challenges we ran into

  • Bundle Terraform state drift. Restructuring the App resource left a stale databricks_app entry in the local tfstate pointing at a workspace App that had already been deleted; the next bundle deploy tried to refresh it and 404'd. Repaired by surgically removing the stale entry from .databricks/bundle/dev/terraform/terraform.tfstate.

  • Workspace's "Git repository required" Apps policy. A purely workspace-uploaded source_code_path was rejected at App-create time. Fixed by attaching git_repository to the App resource and threading the GitHub URL + branch through bundle variables.

  • Streamlit version mismatch on the Apps runtime. st.pydeck_chart(key=..., height=...) works locally on the latest Streamlit but throws TypeError on the deployed runtime. Refactored to set height on the pdk.Deck object and forward key to each pdk.Layer's id parameter — same effect, version-agnostic.

  • Module-import break after source_code_path: app. Locally from app.X worked; in the container the runtime sees /app/python/source_code/{main.py,components/,services/} with no app/ parent. Dropped the app. prefix everywhere, vendored a minimal _shared_config.py so the App tree no longer reaches into pipelines/, and copied the runtime YAML configs into app/config/.

  • Silent empty state filter. SELECT DISTINCT canonical_state AS state … ORDER BY canonical_state failed with UNRESOLVED_COLUMN on Spark — after SELECT DISTINCT, ORDER BY can only see projected aliases. query_df swallowed the error and returned an empty DataFrame, so the sidebar collapsed to just (All India) with no signal why. Fixed in one line by ordering by the alias.

    Accomplishments that we're proud of

  • A trust model that admits its own ignorance. Confidence-as-alpha is the visual differentiator and it's also a contract: a high score with low confidence is meaningfully different from a high score with high confidence, and the map says so.

  • Citations everywhere. Every claim a facility makes propagates through to the Action Center as a quoted snippet from the source text. Auditable by design — see docs/gold_contract_decision.md.

  • Genie binding that just works. Declarative App↔Genie permission via the bundle, with space content provisioned via API; planners type questions in English and get back SQL + tables grounded in the same gold layer.

  • Metadata-driven ETL. Adding a new facility source is editing one YAML; the framework handles the bronze read, the silver normalize, the alias join, the H3 geocode, and the contract.

What we learned

  • Streamlit is fast-to-build, brittle-to-ship. The runtime version on Databricks Apps lags the latest pip release by several minor versions; any kwarg added after that line silently breaks. Lesson: prefer code paths that work across Streamlit 1.30+ rather than relying on the freshest API.
  • Workspace policies shape architecture more than any technical choice. "Git repository required for Apps" is a one-line workspace setting that forced a deployment refactor. Worth surfacing those constraints early rather than discovering them at App-create time.
  • Decide the gold contract first, re-wire never. Picking h3_care_score over medical_desert_* as the canonical UI surface (docs/gold_contract_decision.md) saved us from a full re-wire of services + pages + tooltips when the demand-aware scorer landed later.
  • "Data-deficient" deserves to be a category, not an empty cell. Visually marking weak-evidence cells grey instead of dropping them taught us how often the absence of data is the most actionable signal.

What's next for C3

  • Wider source coverage — push beyond the 10,000-record demo sample to NIN, district-level health-information systems, and state-published facility registers; the metadata-driven framework was built for this.
  • FastAPI surface for programmatic access — let other planning tools embed the trust-weighted scores and citations without going through the UI.
  • Gold tables synced to Lakebase via Lakebase Sync (reverse ETL) so map reads hit Postgres at sub-second latency and the warehouse is reserved for batch.
  • Deeper NACHC root-cause tooling in the Action Center — current root-cause panel categorises gaps as Data / Service / Engagement; the next pass adds intervention recommendations and scenario costing.
  • Feedback loop on trust. Planner overrides currently log the correction; next they recalibrate claim weights so the trust model gets sharper every time a human says "this row is wrong".

Built With

Share this project:

Updates