Inspiration Disaster data in India doesn't arrive in one place — it arrives everywhere at once. Official feeds from IMD and MOSDAC/ISRO sit alongside weather APIs, websites and RSS feeds, public weather datasets, social media posts, and raw citizen reports. During the Kerala floods, Telegram channels became a genuine lifeline for ground-level information, yet that signal was never systematically captured.
That fragmentation creates six compounding problems: data scattered across sources, the same event reported many times over, reports of wildly varying reliability, volumes too high for manual monitoring, critical events buried in the queue, and network outages cutting people off from the alerts they need most.
We were also inspired by the direction of national efforts like IMD/MoES's Multi-Hazard Early Warning Decision Support System and the global Early Warnings for All push. The gap wasn't a shortage of data — it was the absence of a single, trustworthy, prioritized layer on top of it. So we built one.
What it does Data Drishti turns scattered, noisy, duplicated disaster chatter into verified, geotagged, prioritized, real-time intelligence.
It ingests multi-source data into one standardized event format, then uses ML to detect when several reports describe the same event, reconcile conflicts, and fuse them into a single canonical event. Each event carries three explainable scores:
Corroboration Score — weighted event evidence, normalized using only the weights of signals actually available, so a missing feed never silently drags a score down. Trust Index — evidence fusion across source reliability, independent corroboration, physical/weather observation consistency, temporal consistency, location consistency, and content quality. Urgency Index — dynamic priority built from severity, reported immediacy, trust, corroboration, location risk, and an aging factor so nothing rots in the queue. From there, a real-time dashboard (React + Leaflet + Chart.js) gives responders a live geographic picture, while alerts reach citizens over WhatsApp, Telegram, and SMS — the same channels they can use to submit reports back.
A few things we deliberately built in:
Always shows the reason. Every score ships with a plain-language explanation, never a bare number. Priority pipeline. Urgent reports bypass the normal queue. Multi-source verification. Credibility comes from independent source types, not repeated copies of the same claim. Offline access. Last-known disaster data stays available when the network drops. Multilingual by default. Hindi and English supported natively across reporting and alerts. Continuity. Critical operations keep running even if an AI component fails. How we built it The system runs as seven layers: data sources → ingestion & streaming → real-time processing → AI/ML evidence fusion → event prioritization & storage → API & real-time delivery → user platform.
Ingestion & messaging backbone. Apache Kafka buffers, distributes, and partitions incoming events. Cursor checkpoint tracking prevents re-ingestion of data we've already processed.
Stream processing. Apache Flink handles event-at-a-time, stateful matching and context analysis, so related reports are correlated as they arrive rather than in overnight batches. A fast, free triage stage routes critical events into a priority lane.
Evidence Intelligence Engine. Four stages: corroborate, match, fuse evidence, score urgency. Reports A, B, and C collapse into one canonical event with a defensible score behind it.
Storage & delivery. PostgreSQL with PostGIS for geospatial queries and pgvector for similarity matching.
Models. MuRIL for Hindi/English NLP, XLM-R for cross-lingual work, Isolation Forest for weather/sensor anomaly detection, XGBoost for structured prediction and priority signals, and U-Net/CNN for satellite image segmentation.
Infrastructure. Docker for containerization and Kubernetes for orchestration, keeping the whole thing scalable and deployable.
Stack choice. We went 100% open source — it keeps infrastructure and licensing costs low while letting new sources and models plug in without a redesign.
Challenges we ran into Entity resolution across incompatible formats. Deciding when two reports are genuinely the same event — versus two nearby but distinct events — is harder than it looks when sources disagree on naming, granularity, and timestamps.
Scoring fairly when data is missing. Early versions punished events for having fewer available signals. We solved it with a normalization term that divides by the weights of available signals only, so absence of data isn't mistaken for lack of credibility.
Latency versus accuracy. Running heavy verification before showing anything means responders wait. The priority lane exists precisely to break that trade-off for the events that matter most.
Exactly-once ingestion. Without checkpoint tracking, restarting a consumer replayed old reports and inflated corroboration counts artificially.
Graceful degradation. Designing so that an AI component failing degrades the system rather than stopping it took deliberate separation between the critical path and the enrichment path.
Connectivity gaps. Offline access meant treating the client as a cache with reconciliation logic, not a thin display layer.
Privacy. Citizen reports carry location and identity data, so access control had to be designed in from the start rather than bolted on.
Accomplishments that we're proud of A working end-to-end pipeline where fragmented multi-source reports converge into a single canonical, scored event. An explainable scoring engine — corroboration, trust, and urgency each traceable to named inputs and weights, not a black box. Priority routing targeting under 5 ms, so urgent reports genuinely skip the queue. Roughly 97% classification accuracy measured on our current evaluation. Graceful degradation and offline access built in from day one, not added as an afterthought. Native Hindi and English support across both reporting and alerting. A 100% open-source stack, with a modular design that accepts new sources and models without redesign. A Technology Readiness Level of 6 — demonstrated in a relevant environment. What we learned Volume is not the same as signal. The hardest problem in disaster data isn't collecting more of it — it's deciding which reports deserve belief and which deserve attention first.
Explainability turned out to be a functional requirement, not a nice-to-have. Responders won't act on a number they can't interrogate, which is why every score carries its reasoning.
We also learned that streaming systems demand a different mindset from batch ones: statefulness, ordering, replay safety, and checkpoint discipline stop being optimizations and become correctness issues. And designing for failure — network loss, component crashes, source outages — forced us to be far more deliberate about what sits on the critical path.
Finally, building for both authorities and citizens taught us that one verified data layer can serve very different consumers: farmers needing advisories, responders needing locations, researchers and media needing reliable records, and citizens needing to know whether the water is rising on their street.
What's next for National Weather Big Data Analytics Platform Benchmark priority routing against the sub-5 ms target under realistic production load, and publish the results. Broaden the source base — additional satellite feeds, state-level sensor networks, and regional-language social sources. Expand beyond Hindi and English, prioritizing languages spoken in high-hazard coastal and riverine districts. Pilot with disaster management authorities, feeding verified event streams into existing early-warning workflows rather than replacing them. Close the learning loop — use responder confirmations and rejections to retrain matching and trust models continuously. Strengthen offline-first clients so last-known data, queued reports, and reconciliation work reliably on intermittent connections. Scale the deployment on Kubernetes to absorb monsoon-season surge volumes without redesign.
Built With
- apache
- docker
- flink
- kafka
- kubernetes
- machine-learning
- natural-language-processing
- postgis
- postgressql
- python
Log in or sign up for Devpost to join the conversation.