EarthSense AI - Hackathon Project Submission Write-up
๐ฟ Project Summary
EarthSense AI is a production-ready, explainable AI platform that detects global deforestation from satellite frames. By combining a lightweight deep learning classifier (MobileNetV2) with professional remote sensing spectral indices (VARI, GLI, NGRDI) and an interactive Leaflet GIS Explorer, EarthSense converts raw coordinates into georeferenced, explainable, and actionable environmental audits.
๐ก Inspiration
Deforestation is responsible for roughly 15% of global greenhouse gas emissions and rapid biodiversity loss. While satellite imagery is widely available, current environmental monitoring tools suffer from two critical limitations:
- The "Black Box" Problem: AI models output binary predictions (e.g., "90% deforested") without explaining why, leaving field officers and judges unable to verify the audit.
- Lack of Telemetry Integration: Static file uploaders fail to provide geographic context, isolating predictions from actual field operations.
We were inspired to build EarthSense AI Version 2.0 to bridge this gap, creating a transparent, explainable satellite intelligence dashboard that links deep learning directly with real-world GIS coordinates.
โ๏ธ What it Does
EarthSense AI provides a full-suite satellite diagnostic dashboard:
- GIS Satellite Map Explorer: An interactive dark-mode Leaflet map allows users to click anywhere on Earth to initiate a simulated satellite downlink. The app fetches the real, live satellite photography of those coordinates from Esri's global servers in real-time.
- Deep Learning Deforestation Classifier: A MobileNetV2 model classifies the frame as Deforested or Not Deforested in under 2 seconds.
- Interactive Grad-CAM Compare Slider: Generates a gradient-weighted heatmap showing exactly where the neural network focused (e.g., identifying grid logging roads or clearings). An interactive curtain slider lets users wipe between the raw satellite image and the heatmap.
- Spectral Remote Sensing Telemetry: Calculates professional vegetation indices (VARI, GLI, NGRDI) in parallel using red, green, and blue light bands to estimate canopy health and chlorophyll density.
- GPS EXIF Parsing: Automatically extracts embedded coordinates from geotagged satellite JPEGs, mapping them on OpenStreetMap.
- Parallel Batch Pipeline: Allows users to upload multiple frames simultaneously, displaying a pie chart distribution of forest damage and exporting structured CSV reports.
- Auditing Reports: Compiles the complete telemetry (coordinates, indices, heatmaps, and predictions) into a printable PDF report.
๐ ๏ธ How We Built It
We built EarthSense AI with a modern, production-grade architecture:
โ๏ธ Frontend
- React (Vite): Powering a responsive, glassmorphic dark-mode dashboard.
- Leaflet JS: Providing mapping rendering and custom sensor pin interactions.
- Axios: Managing API requests to the Python backend.
- Tailored CSS: Crafted from scratch to provide sleek animations and the hardware-accelerated curtain slider clip-path.
๐ Backend
- Flask (Python 3.10): A lightweight API serving single, batch, and ping routes.
- Gunicorn: Serving production requests in a single-worker environment.
๐ง Machine Learning & GIS
- TensorFlow & Keras 3: Loading and running inference on our custom-trained MobileNetV2 model.
- Numpy & Pillow: Running the mathematical equations for VARI, GLI, and NGRDI spectral indices.
- OpenCV: Generating the jet-color Grad-CAM heatmaps.
- Esri MapServer API: Dynamically fetching $1\text{km} \times 1\text{km}$ static satellite imagery tiles from coordinates.
๐ง Challenges We Overcame (The Engineering Grit)
Building a deep learning platform on cloud servers with limited free-tier resources led to several complex engineering bottlenecks:
- The 512MB RAM Constraint (Render OOM):
TensorFlow is notoriously heavy. Initializing TF and loading the model exceeded Render's 512MB RAM limit, causing OOM crashes.
- Solution: We disabled TensorFlow's heavy CPU calculation cache (
oneDNNops) programmatically and restricted Gunicorn threads to1to eliminate multi-threaded memory overhead. We also skip Keras optimizer loading (compile=False) and run Python garbage collection (gc.collect()) immediately after the model loads, saving over 150MB of RAM and keeping startup memory under 350MB.
- Solution: We disabled TensorFlow's heavy CPU calculation cache (
- Gunicorn Process Deadlocks:
Under Gunicorn, Keras's standard
model.predict()allocates background thread queues that conflicted with Gunicorn's thread-pool workers, causing requests to hang indefinitely.- Solution: We refactored prediction endpoints to run direct model tensor calls (
model(x, training=False).numpy()). This executes inference synchronously in the request thread, bypassing dataset queues and resolving the deadlock.
- Solution: We refactored prediction endpoints to run direct model tensor calls (
- Keras 2 vs. Keras 3 Mismatches:
The pre-trained model was saved in Keras 3 (which uses
batch_shapelayers), but Render's python default package installed Keras 2 (which usesbatch_input_shape), causing deserialization crashes.- Solution: We forced Python version
3.10.13in standard UTF-8runtime.txtand pinnedtensorflow-cpu>=2.16.0to force Keras 3 alignment.
- Solution: We forced Python version
- Pillow RGBA to JPEG Encoding OSError:
Our transparency-blended Grad-CAM heatmap was generated as an
RGBAimage. Pillow crashed with anOSErrorwhen attempting to save the transparency layer as aJPEGbase64 string.- Solution: We converted the blended image back to
RGBmode prior to base64 encoding.
- Solution: We converted the blended image back to
๐ Accomplishments We're Proud Of
- Live Satellite Downlinks: Clicking anywhere on the map pulls a real satellite photo of that coordinate and runs prediction on itโno mocks needed!
- Seamless Curtain Compare UX: Implementing the compare slider purely in hardware-accelerated CSS, providing smooth gestures.
- Resource Optimization: Fitting a heavy TensorFlow model into a tiny 512MB container with instant response times.
- ใขใฏใทใงใณๅฏ่ฝใช PDF Reports: Generating professional, printable audit reports.
๐ What We Learned
- Keras 3 API Lifecycle: Learned to manage changes in input layers and model imports.
- Geospatial Web Mapping: Learned how Leaflet manages projections and coordinates mapping.
- Server-side profiling: Gained experience profiling memory usage on cloud instances.
๐ฎ What's Next for EarthSense AI
- Multispectral Band Support (Sentinel-2): Integrate Near-Infrared (NIR) bands to calculate NDVI (Normalized Difference Vegetation Index) for cloud-penetrating vegetation classification.
- Edge Deployment (TensorFlow Lite): Compile the MobileNetV2 model to TFLite for edge deployment on low-cost drones.
- Real-time Alert System: Set up automated cron jobs to scan selected coordinate grids daily and send automated alerts to park rangers when forest cover drops below a set threshold.
Built With
- esri
- flask
- gunicorn
- imagery
- leaflet.js
- opencv
- react
- satellite
- tensorflow
- world
Log in or sign up for Devpost to join the conversation.