Krishi-Intel

Krishi Intel

Comment

🌾 Krishi-Intel: Autonomous Agri-Procurement & Mandi Price Surge Engine

Python 3.10+ DuckDB Pandera LightGBM Google Gemini ReportLab License: MIT

An end-to-end, enterprise-grade AI procurement intelligence engine designed for Indian Quick-Commerce networks (Blinkit, Zepto, Swiggy Instamart) and FMCG supply chains.

Krishi-Intel ingests uncurated Agmarknet APMC transaction records[cite: 3], enforces schema contracts[cite: 3], standardizes entity anomalies with fuzzy clustering[cite: 3], forecasts multi-horizon commodity price corridors with Quantile Machine Learning[cite: 3], optimizes inter-mandi spatial freight arbitrage[cite: 3], and triggers an autonomous Google Gemini 2.5 Flash agent to compile executive B2B strategy directives and downloadable PDF reports[cite: 1, 3].

🏛️ System Architecture


┌────────────────────────────────────────────────────────────────────────────────────────┐
│                                   KRISHI-INTEL PIPELINE                                │
└────────────────────────────────────────────────────────────────────────────────────────┘
│

1. RAW DATA INGESTION & QUALITY CONTRACTS (DuckDB + Pandera)
├── 811k+ Uncurated APMC Records Ingestion
└── Pandera Schema Contracts & Type Assertion
│


2. DIRTY DATA CLEANING & ENTITY RESOLUTION (RapidFuzz)
├── Fuzzy String Matching on 28 States & 1,500+ Mandis
├── Datetime Multi-Regex Standardizer
└── Physical Price Bound Rectification (Min <= Modal <= Max)
│


3. FEATURE ENGINEERING & DOMAIN METRICS
├── Grade Dispersion Ratio (GDR) Spread Tracking
├── Temporal Lags (1d, 3d, 7d, 14d, 28d) & Rolling Volatility (CV)
└── Monsoon & Festive Agri-Calendar Flags
│


4. MULTI-HORIZON QUANTILE FORECASTING & SHAP (LightGBM)
├── Multi-Quantile Regression (P10 Bear Floor, P50 Median, P90 Surge Ceiling)
├── 25% Supply Shock Risk Classification
└── TreeSHAP Global & Local Attribution
│


5. SPATIAL ARBITRAGE & FREIGHT ECONOMICS ENGINE
├── Road Transit Modeling (Rs 4.5/ton-km)
├── Perishable Spoilage Buffers (4%)
└── Landed Cost Optimization vs. Spot Market Rates
│


6. AUTONOMOUS GENAI COPILOT & PDF COMPILER (Gemini 2.5 Flash + ReportLab)
├── Dynamic LangChain Function Calling (`query_forecast`, `calculate_arbitrage`)
├── Multilingual Dispatch Generation (Hindi, Tamil, Telugu, Kannada, Marathi)
└── Automated Executive C-Suite PDF Report Generation

🚀 Key Features

  • Out-of-Core SQL Ingestion: Processes 811,131 raw transactions seamlessly using embedded DuckDB with minimal RAM overhead[cite: 3].
  • Automated Data Quality & Schema Contracts: Uses Pandera for strict upstream validation and RapidFuzz string clustering to resolve multi-dialect spelling corruptions across Indian states and APMCs[cite: 3].
  • Probabilistic Price Corridors: Multi-Quantile LightGBM models ($\text{P}{10}, \text{P}{50}, \text{P}_{90}$) replace point predictions with risk corridors, achieving a 13.15% MAPE[cite: 3].
  • Spatial Freight Optimization: Models diesel transit economics ($\text{Rs } 4.5/\text{ton-km}$) and transit spoilage shrinkage ($4\%$) to uncover inter-state price arbitrage[cite: 3].
  • Agentic GenAI Sourcing Director: Integrates Google Gemini 2.5 Flash with function-calling capabilities to evaluate live market data, deliver multilingual WhatsApp dispatches, and build executive PDF directives[cite: 1, 3].

📊 Empirical Business Impact Simulation

In a simulated 50-Metric Ton (500 Quintal) Tomato procurement order for Delhi NCR fulfillment centers during a localized supply crunch[cite: 1]:

Metric Local Spot Procurement Krishi-Intel Spatial Route (MH/MP) Business Impact / Net Delta
Procurement Base Price Rs 3,450.00 / qtl[cite: 1] Rs 2,100.00 / qtl[cite: 1] -39.1% Base Rate
Transit Freight & Spoilage Rs 0.00 / qtl Rs 691.50 / qtl[cite: 1] Rs 607.50 Freight + 4% Spoilage[cite: 1, 3]
Effective Landed Rate Rs 3,450.00 / qtl[cite: 1] Rs 2,791.50 / qtl[cite: 1] Landed Price Optimization[cite: 1]
Total Sourcing Outlay INR 1,725,000 INR 1,395,750[cite: 1] INR 3,29,250 Cost Reduction[cite: 1]
Margin Protection Baseline +19.3% Net Savings[cite: 1] Hedging against 26.8% surge[cite: 1]

🛠️ Repository Layout


krishi-intel-engine/
├── data/
│   └── Agriculture_price_dataset_unclean_full_more_messy.csv   # Kaggle dataset
├── pipelines/
│   ├── ingestion_duckdb.py      # DuckDB streaming & Pandera schema models
│   └── data_cleaning.py         # RapidFuzz entity resolution & price rectification
├── models/
│   ├── feature_engineering.py   # Temporal lags, rolling CV, and agri-seasonality
│   ├── quantile_lightgbm.py     # Multi-quantile LightGBM training & validation
│   └── shap_explainability.py   # TreeSHAP feature importance plots
├── genai_copilot/
│   ├── agent.py                 # LangChain + Gemini 2.5 Flash tool-calling agent
│   └── pdf_generator.py         # ReportLab automated PDF report compiler
├── outputs/
│   ├── Krishi_Intel_Procurement_Directive.pdf  # Auto-generated C-Suite PDF report
│   └── shap_summary.png         # SHAP importance visualization
├── requirements.txt
└── README.md

⚡ Quickstart & Installation

1. Clone the Repository

git clone [https://github.com/your-username/krishi-intel-engine.git](https://github.com/your-username/krishi-intel-engine.git)
cd krishi-intel-engine

2. Install Dependencies

pip install -r requirements.txt

Required packages:

duckdb>=0.10.0
pandera>=0.18.0
rapidfuzz>=3.6.0
lightgbm>=4.3.0
shap>=0.45.0
langchain>=0.2.0
langchain-google-genai>=1.0.0
reportlab>=4.1.0
pandas>=2.0.0
matplotlib>=3.8.0
seaborn>=0.13.0

3. Configure API Credentials

Set your Google Gemini API key:

export GOOGLE_API_KEY="your-gemini-api-key-here"

4. Run the Pipeline

Execute the end-to-end Python pipeline or open the notebook in Google Colab:

python main.py

📈 Machine Learning Performance & Validation

  • Validation Strategy: Out-of-time temporal split (Train: 80% chronological historical data, Test: 20% holdout window).

  • Evaluation Metrics:

  • $\text{P}_{10}$ Pinball Loss: 54.48

  • $\text{P}_{50}$ Pinball Loss: 112.68

  • $\text{P}_{90}$ Pinball Loss: 56.47

  • $\text{P}_{50}$ Median MAPE: 13.15%

  • Key SHAP Feature Drivers: daily_modal (previous price), rolling_mean_7d, lag_1d, rolling_mean_14d, and grade_dispersion_ratio.

📄 Auto-Generated PDF Report Sample

The ReportLab engine compiles execution summaries into a formatted PDF:

+-------------------------------------------------------------------------------+
|  KRISHI-INTEL | AUTONOMOUS PROCUREMENT DIRECTIVE                              |[cite: 1]
|  Supply Chain Risk Assessment, Quantile Forecasts & Inter-Mandi Arbitrage     |[cite: 1]
+-------------------------------------------------------------------------------+
|  Target Commodity : Tomato (50 MT)    |  Destination : Delhi NCR Fulfillment  |[cite: 1]
|  Surge Risk       : HIGH SHOCK RISK   |  Net Savings : INR 329,250 (19.3%)    |[cite: 1]
+-------------------------------------------------------------------------------+
|  1. Quantile Pricing Corridor (P10: Rs 2400 | P50: Rs 3450 | P90: Rs 4200)    |[cite: 1]
|  2. Spatial Arbitrage: #1 Maharashtra (Landed Rs 2791.50 vs Local Rs 3450.00)|[cite: 1]
|  3. Executive B2B Strategy & Hedging Directive (Gemini 2.5 Flash)             |[cite: 1]
+-------------------------------------------------------------------------------+

📑 Dataset Attribution

This project utilizes the Indian Agricultural Mandi Prices Dataset (2023–2025) sourced from Agmarknet / Ministry of Agriculture and Farmers Welfare, compiled by Arun Kumar Giri on Kaggle:

📜 License

Distributed under the MIT License. See LICENSE for more information.

Built With

  • jupyter-notebook
Share this project:

Updates