CropVar-MM

CropVar-MM: SoC-Enabled Edge-Deployed Multimodal CNN–LLM Framework for Crop Variety Identification from Satellite Data and Agronomic Decision Support

Comment

🌱 Inspiration

Agriculture is the livelihood of over 50% of India’s population, but smallholder farmers face:

Fragmented land and limited mechanization.

Unpredictable climate variability (droughts, floods, pests).

Lack of access to AI-driven decision support due to poor connectivity and high infrastructure costs.

We were inspired to bridge the gap between AI research and rural farming adoption. CropVar-MM was born to ensure that AI reaches farmers in their fields, in their language, on affordable devices.

🌾 What it does

Classifies multiple crop varieties using multispectral + SAR satellite imagery.

Generates farmer-friendly recommendations (irrigation, fertilization, pest control) using an LLM.

Works offline on System-on-Chip (SoC) edge devices and NVIDIA Jetson.

Provides localized insights in regional languages (Tamil, Hindi, etc.).

Supports policymakers with yield forecasting and sustainability tracking.

🛠 How we built it

Data Collection

Used Sentinel-1 (SAR), Sentinel-2 (multispectral), and ResourceSAT imagery.

Preprocessing

Cloud masking, normalization, temporal aggregation.

Model Training

CNN for crop classification (pixel-level segmentation).

LLM for translating crop maps into actionable farmer advice.

Optimization for Edge

Applied quantization and pruning to reduce computation.

Deployed on Jetson Nano and Raspberry Pi 5.

User Interface

Multilingual mobile app with voice + text support.

⚡ Challenges we ran into

Cloud cover issues in optical data → solved using SAR fusion.

Limited labeled datasets → used transfer learning and augmentation.

Edge hardware limitations → pruning + TensorRT optimization.

Designing farmer-friendly outputs → simplified AI insights into natural language.

Time constraints during hackathon → had to prioritize efficiency over extra features.

🏆 Accomplishments that we're proud of

Built a working prototype of an end-to-end multimodal pipeline.

Achieved high classification accuracy on AgriFieldNet benchmark crops.

Successfully ran inference on a Jetson Nano with <1s latency.

Delivered outputs in local languages, ensuring accessibility.

Formed a multidisciplinary team bridging AI, agriculture, and embedded systems.

📚 What we learned

The power of multimodal AI when combining satellite imagery and LLMs.

Techniques for model compression on SoC devices.

Importance of human-centered AI design for adoption.

Practical exposure to real agricultural datasets like AgriFieldNet.

Value of collaboration and time management in hackathons.

🚀 What's next for CropVar-MM

Scale pilot testing in Thanjavur and surrounding agricultural regions.

Add real-time IoT sensor integration (soil moisture, pH).

Implement federated learning to improve models without sharing farmer data.

Partner with Krishi Vigyan Kendras (KVKs) and NGOs for field-level deployment.

Expand to South Asia and Africa with crop-specific fine-tuning.

Integrate Generative AI downscalers for high-resolution climate predictions.

Share this project:

Updates