EARTHZI

Seeing Earth in every way that matters.

TRY IT OUT - https://relandhruvi.github.io/EARTHZI

Backend API - https://relandhruvi-earthzi.hf.space

GitHub - https://github.com/relandhruvi/EARTHZI

Presentation - https://drive.google.com/file/d/1zW7f2-1jtAjvrPsGIUeE1p2jkCI1lVu9/view?usp=sharing

Inspiration

Every second, satellites orbit above us—Sentinel-1, Sentinel-2, and countless others—capturing the same slice of Earth in fundamentally different ways. An optical sensor shows us what we see with our eyes. Synthetic Aperture Radar (SAR) pierces through clouds and darkness. Multispectral cameras reveal what's invisible to human vision.

But here's the tragedy: these perspectives remain isolated from each other.

When a monsoon floods agricultural lands in India, disaster response teams are trapped in silos. An SAR image could detect the exact extent of the flooding, but they're searching metadata from years past. When researchers track deforestation, they manually cross-reference images from different sensors, burning countless hours. When governments need real-time intelligence about rapidly changing regions, they're blocked by the sheer volume of archived data.

We asked ourselves a fundamental question: What if we could teach machines to see across these different perspectives the way human experts do? What if a field team in Bangladesh could point at a satellite image of their region and instantly find every similar location in the entire archive—regardless of sensor type, season, or weather?

EARTHZI exists because the Earth cannot wait for manual processes. Our climate is changing. Disasters don't announce themselves. Communities need answers, not delays.

What It Does

EARTHZI is a cross-modal satellite image retrieval engine that bridges the gap between different sensor modalities.

Upload an image. Find the world.

Bidirectional retrieval: Upload SAR and get optical matches. Upload optical and get SAR. One model, both directions.
Sub-second search: Comb through 4,000+ real satellite scenes in milliseconds. Scale seamlessly to 100,000+ images as needs grow.
Geographic understanding: EARTHZI doesn't just match pixels—it understands that the same place looks different under different conditions. It learns semantic relationships, not surface-level patterns.
Real retrieval, real data: Every image in our demo is from Sentinel-1 and Sentinel-2. No synthetic training data. No mock datasets. This is the archive that matters.

Use cases that are already saving time:

Disaster response: Find flooded regions across sensor modalities in seconds
Climate monitoring: Track deforestation, land-use change, and glacial retreat
Agricultural assessment: Monitor crop health and predict yields before harvest
Urban planning: Detect informal settlements and infrastructure changes
Environmental conservation: Monitor protected areas for illegal activity

How We Built It

We tackled one of the hardest problems in machine learning: teaching a single model to understand two fundamentally different ways of seeing the world.

Stage 1: Fine-tuning RemoteCLIP We took a vision-language model designed for satellite imagery and adapted it specifically for paired Sentinel-1 and Sentinel-2 imagery. Instead of generic pretraining, we fine-tuned on thousands of precisely aligned SAR-optical pairs, teaching the model cross-modal relationships it would never learn from single-modality data.

Stage 2: Contrastive Learning at Scale We implemented a contrastive learning framework that pulls SAR-optical pairs closer in embedding space while pushing unrelated images apart. This creates a shared semantic representation where proximity equals similarity—regardless of which sensor captured the image.

Stage 3: Retrieval Infrastructure Built on FastAPI, our backend serves embeddings at production speed. We generate a 512-dimensional embedding for each satellite scene in milliseconds, then use cosine similarity search to rank results. Future iterations will scale to millions using FAISS vector databases and Pinecone.

Frontend that tells the story: Three.js particle animations and GSAP scroll triggers create a cinematic experience. Geographic data visualization shows you exactly where matches appear on Earth. The interface isn't just functional—it's built to make the discovery process feel like exploration.

Full stack deployment: Hugging Face Spaces hosts the model. GitHub Pages serves the frontend. FastAPI endpoints handle the backend inference. Everything tied together with automated CI/CD.

Challenges We Ran Into

1. The modality gap is real SAR and optical images of the same place can look almost nothing alike. SAR shows you backscatter and rough texture. Optical shows color and reflectance. We spent weeks experimenting with different loss functions before discovering that contrastive learning—pulling similar pairs together—was the key to bridging this gap.

2. Data alignment headaches Getting perfectly aligned Sentinel-1 and Sentinel-2 pairs proved harder than expected. The sensors have different resolutions, different acquisition times, different geometric properties. We had to build custom preprocessing pipelines and interpolation scripts just to create training data.

3. Computational constraints Fine-tuning a vision model on thousands of image pairs while iterating quickly required serious optimization. We compressed models, used mixed precision training, and distributed inference across multiple GPUs.

4. Scaling the database 4,000 images sounds small until you're doing cosine similarity search. We profiled bottlenecks, implemented batch processing, and designed our embedding pipeline for horizontal scaling.

5. Making it intuitive The user interface needed to make cross-modal retrieval feel natural, not academic. Early prototypes felt like academic papers. We redesigned to emphasize geographic discovery and immediate understanding.

Accomplishments We're Proud Of

✨ A working cross-modal retrieval system that actually solves a real problem in Earth observation

✨ Real data throughout: Every image in our live demo comes directly from Sentinel satellites. No cheating with synthetic or curated datasets.

✨ Sub-second search: Search 4,000 images and get ranked results in under 1 second. This isn't theoretical—it works in the browser, right now.

✨ Bidirectional architecture: One model. Both directions. Zero manual effort to swap modalities. This elegance took weeks to achieve.

✨ A cinematic interface: Three.js particle effects and GSAP scroll animations that make satellite imagery exploration feel like something from a sci-fi film, not a research paper.

✨ End-to-end pipeline: From satellite data ingest to live retrieval, this is a complete, deployable system—not a proof-of-concept.

✨ Community-first revenue model: We built multiple pathways to sustainability (government contracts, research grants, institutional licensing) that don't compromise the free tier for NGOs and researchers.

What We Learned

1. Cross-modal learning is solvable with the right approach We tried several architectures before landing on fine-tuned RemoteCLIP + contrastive learning. The lesson: sometimes the simplest approach is the right one.

2. Real data beats perfect data We could have spent months creating the perfect synthetic training dataset. Instead, we aligned real satellite pairs. Real beats perfect every time.

3. Earth observation experts think differently than ML researchers When we showed early results to geographers and disaster response teams, their feedback completely redirected our roadmap. We weren't building for data scientists—we were building for people trying to save lives.

4. Interface clarity matters more than feature completeness Our first prototype had 20 filters and options. The version that got traction had 3 buttons and a simple upload. Constraints drive clarity.

5. Deployment is half the battle A model trained in isolation is useless. Building the infrastructure to serve embeddings at scale, cache results, and handle concurrent requests was as important as the ML work.

What's Next for EARTHZI

Short Term (3 months)

Expand the gallery from 4,000 to 100,000+ images with geographic metadata (coordinates, season, biome)
Add Optical→SAR bidirectional retrieval (currently perfecting SAR→Optical)
Geographic filters so users can search "show me all matches in monsoon regions"

Medium Term (6-12 months)

Vector database (FAISS / Pinecone) for million-scale search
"Text-to-satellite" search: "Show me flooded agricultural land" → instant results
Mobile-friendly interface for field teams in remote regions
Integration with QGIS and other GIS platforms

Long Term (12+ months)

Real-time retrieval from live Sentinel satellite feeds
Multi-modal fusion: SAR + Optical + Thermal data in a single embedding space
ISRO archive integration: Access India's entire satellite history
Plug-in for Google Earth Engine and QGIS so EARTHZI becomes infrastructure, not just an app

The bigger vision: We're building the search engine for our planet. In 5 years, EARTHZI should be as essential to Earth observation as Google Search is to the internet—the default place scientists, governments, and NGOs go to understand what's happening on the ground.