-
-
Home Screen
-
Minimal Dashboard
-
Prediction Clock
-
Hindi Dashboard on Android Phone
Inspiration
Street vendors are one of the most heat-exposed groups of workers in India, and one of the least served by existing technology. Heatwave don't always announce themselves and the damage they cause is quiet, but possibly devastating. One vendor we spoke to misjudged how hot it actually was. By the time he noticed, his produce had already spoiled, sitting in full sun while he waited out what he thought was a manageable afternoon. That was a couple day's worth of livelihood that could have been protected way better.
We noticed that the infrastructure gap is closing faster than people realise. Affordable smartphones and data plans mean more vendors are connected than ever. UPI has transformed how they earn, making digital payments straightforward and giving millions a stable livelihood. Our app is built to protect that livelihood. If they already have a phone and a way to get paid, Canopy gives them a way to make smarter decisions about when to show up, what to watch for, and when to pack up early.
What It Does
A vendor opens Canopy, taps one button, and selects what they're selling. Within seconds they get a heatwave risk assessment tailored to their specific inventory. Not just a generic temperature reading, but actionable guidance. A produce vendor gets different advice than a clothes seller or someone running a fried snacks stall.
Every assessment includes an urgency level and two concrete mitigation steps. It's delivered in the vendor's regional language across nine Indian languages — Hindi, Tamil, Telugu, Bengali, Marathi, and more. This is read aloud, so vendors with low text literacy or busy hands can still act on it. The app is installable directly from the browser and designed to work on mid-range Android devices, the phones vendors in India actually carry.
The AI system powering this is a multimodal retrieval pipeline we call Visual RAG. It pulls fine-grained meteorological data from NOAA HRRR, generates a spatial heat intensity map of the vendor's area, and uses Amazon Nova Multimodal Embeddings to find the most similar scenario from our historical heatwave database. That retrieved context, combined with live temperature and heat index data from Open-Meteo, goes to Amazon Nova 2 Lite, which reasons over all of it to produce a structured, inventory-aware risk assessment.
How We Built It
Canopy runs on a two-layer architecture. The primary path is a FastAPI backend deployable as AWS Lambda: it pulls high-resolution gridded weather data from NOAA HRRR, renders a spatial heat intensity map for the vendor's location, and feeds that image into Amazon Nova Multimodal Embeddings to retrieve the closest match from a curated database of IMD-linked historical heatwave events, each record containing a 1024-dimensional embedding and a detailed impact-and-mitigation narrative. That retrieved context, alongside live heat index data from Open-Meteo, goes to Amazon Nova 2 Lite for final risk reasoning. The result is structured, inventory-specific, and returned end-to-end in under four seconds (estimated).
If the primary backend is unavailable, a deterministic rule engine in the Next.js layer takes over, which still pulls live weather variables from Open-Meteo, and returns inventory-specific guidance, just without the retrieval layer. The transition is invisible to the user.
The frontend is a Next.js PWA with Workbox-powered caching and a complete Web Push pipeline: subscriptions, server-side fan-out via DynamoDB, and live weather event triggers. Vendors receive proactive heatwave alerts before conditions turn dangerous, not just on demand.
Language support required its own infrastructure. Browser-native speech synthesis silently fails on Indic scripts on most mid-range Android devices. So we built a server-side audio proxy that fetches synthesized speech from Google TTS and streams it through a hidden audio element, bypassing the browser engine entirely. Every element of the UI translates dynamically across all nine languages. Hindi output was reviewed and appreciated by vendors we spoke to directly during testing.
Challenges We Ran Into
Vercel's deployment constraints ruled out the Python packages our full pipeline depends on. We responded by building a lightweight Edge API fallback that delivers real, inventory-specific guidance using live weather variables, so vendors always get actionable output regardless of backend state. This constraint made us appreciate the contributions of the AI layer, and that clarity improved the whole system.
The second challenge was language support on mobile. Getting AI-generated heatwave guidance read aloud in Hindi or Tamil on a ₹8,000 Android phone sounds straightforward. Silent failures in the browser's speech engine on Indic scripts are widespread and underdocumented given the size of that user base. The server-side audio proxy we built to solve this is now one of the more robust parts of the stack.
Accomplishments That We're Proud Of
The Visual RAG pipeline is the thing we're most proud of architecturally. Using HRRR-derived heat maps and multimodal embeddings to retrieve historically similar heatwave scenarios, and grounding Nova's reasoning in documented real-world outcomes rather than just live weather statistics, is a genuinely novel application of retrieval-augmented generation to a hyperlocal, high-stakes decision context.
We're also proud of the end-to-end language experience. Nine languages, fully dynamic, with audio that actually works on the devices our target users own. Vendor feedback on the Hindi output was positive, and getting that reliability on mid-range hardware took real effort. The four vendors we spoke to (two during research, two during testing) shaped almost every UX decision we made.
What We Learned
Building a reliable fallback forced us to reason precisely about what the retrieval layer was actually contributing versus what a well-designed rule engine could do on its own. That distinction sharpened the whole architecture.
The bigger learning was about who you're building for. Talking to vendors changed how we thought about the product. The language support, the audio readout, the one-tap flow felt mandatory to implement after those conversations. The best technical decisions came directly from trying to imagine a vendor with a busy stall, a noisy street, and twenty seconds to decide whether to pack up.
What's Next for Project Canopy
The most immediate priority is expanding the historical heatwave database, which means more IMD-linked events, better regional coverage, richer per-record metadata. That's what raises the ceiling on what the retrieval layer can contribute. Alongside that, we want to fine-tune the risk model on regional heat patterns specific to Indian cities, where the relationship between temperature, humidity, and inventory damage varies significantly.
Longer term, we want to run a real pilot with vendors. The core insight doesn't go away after a hackathon: this demographic is increasingly reachable via smartphone and deeply underserved by existing weather tools.
Log in or sign up for Devpost to join the conversation.