Slide 1: Intro
Slide 2: Systemic Market Asymmetry & Value Destruction
Architecture Diagram
Slide 3: Optimal Terminal Routing & Arbitrage
Slide 4: Impact and Infrastructure
Slide 5: Thank You

KilimoAgent: Autonomous Multimodal Agricultural Arbitrage & Logistics Dispatch Engine

Inspiration

This project started from a reality close to home. My father is an agronomist engineer, and growing up around his work, I saw firsthand what happens after the harvest. Farmers can work the land for months, produce great quality yields, dry and bag their crops, but once the harvest is ready, an entirely different struggle begins.

In Sub-Saharan Africa and across the East African Community (EAC), smallholder farmers lose anywhere between 25% and 35% of their potential income. The reason isn't poor farming; it is acute market information asymmetry, local middleman monopolies, and fragmented rural logistics. When you are standing in a rural depot, you have no transparent way to verify spot commodity rates 200 kilometers away across border trade terminals, let alone book a reliable freight carrier on the spot.

Most agricultural tech platforms try to solve this with complex dashboards, web forms, and typing-heavy interfaces. But that design completely misses the field reality. Farmers in rural areas communicate through spoken languages like Swahili and French, and they judge crops by sight and touch.

I built KilimoAgent to remove that human friction entirely. The goal was to build a true autonomous agent: one where a farmer simply drops a voice note and a photo of their harvest, and the agent takes care of the heavy lifting in the background—auditing the data, grading the crop, calculating net mathematical arbitrage across regional markets, and autonomously issuing a binding freight waybill.

What it does

KilimoAgent operates as an autonomous, zero-text-input execution pipeline:

Multimodal Field Intake: Ingests raw harvest images and spoken voice notes (Swahili, French, English) without requiring mandatory keyboard forms.
Autonomous Visual Grading: Identifies crop species (Zea mays, Cassava, Arabica Coffee, etc.), verifies kernel integrity, assesses moisture levels, and classifies quality into international standards (Grade A / B).
Real-Time Market Arbitrage: Evaluates spot prices across regional trade corridors and calculates net farmer take-home pay using deterministic financial modeling:

$$\text{Net Revenue} = (\text{Volume} \times \text{Spot Price}) - \text{Freight Cost}$$

Autonomous Carrier Dispatch: Locks the optimal terminal, assigns a verified logistics fleet, and generates a binding digital waybill with unique tracking (KILIMO-WB-...).
Generative UI & Conversational Guidance: Features interactive GenUI cards, audio recording toggles, multi-currency display (USD, KES, CDF, UGX, RWF), and on-demand PDF waybill exports.

How we built it

The architecture is decoupled into a resilient multi-model backend deployed on Google Cloud and a modern, accessible frontend.

1. The Dual-Model Backend & Security Pipeline

Orchestration Core (gemini-3.6-flash): Handles multimodal reasoning, native Swahili audio transcription, harvest vision grading, and deterministic tool orchestration via Google GenAI SDK / Agent Development Kit (ADK).
Neural Guardrail Armor (gemma-2-9b-it): Serves as an upstream security interceptor. Before any user payload hits the orchestrator, Gemma audits the input for prompt injections, adversarial overrides, or malicious data poisoning, instantly locking compromised sessions.
Persistent State Machine (Cloud Firestore): Acts as the system memory bank, tracking immutable transaction states (INITIALIZED -> GUARDRAIL_AUDITED -> RUNNING_PIPELINE -> COMPLETED) and maintaining historical dispatch ledgers per farmer.
Infrastructure (Google Cloud Run): The FastAPI backend is fully containerized with Docker and hosted serverless on Cloud Run in us-central1, providing sub-4-second end-to-end execution.

2. The Modern Frontend & GenUI

Stack: Built with React, Vite, and Tailwind CSS using a clean, dark-mode visual hierarchy and Urbanist typography.
Interaction Models: Supports both an instant single-prompt flow and a guided 5-step modal stack. The UI dynamically renders interactive commodity cards, origin selectors, and real-time ledger outputs.
Waybill Ledger & Export: Integrated slide-over drawer connected to Firestore allowing farmers to review past trade settlements and download official PDF waybills.

Challenges we ran into

Building an autonomous agent that bridges raw physical media with financial execution came with several technical hurdles:

Dialect & Audio Nuances: Parsing raw spoken Swahili containing localized rural accents and mixed phrases required careful prompt engineering and grounding to ensure exact weight and location extraction without translation artifacts.
Balancing Autonomy with Determinism: LLMs can be unpredictable with financial calculations. I had to strictly constrain the model's function calling so that revenue and freight math remained 100% deterministic while letting the model reason freely over qualitative image aspects.
State Management in Cloud Run: Handling asynchronous multimodal execution while keeping a clean, verifiable audit trail in Firestore meant engineering a strict state machine to prevent race conditions during rapid retries.
Geospatial & Edge Constraints: Incorporating live distance metrics in areas where mapping APIs have unmapped rural feeder roads required building reliable fallback routing models.

Accomplishments that we're proud of

True Zero-Text Multimodal UX: Achieving complete end-to-end trade execution purely from an unstructured Swahili voice note and an image file, eliminating all mandatory text entry fields.
Robust Dual-Model Security Architecture: Successfully implementing gemma-2-9b-it as an inline security armor that intercepts prompt injections and payload manipulation before reaching the core orchestrator.
Sub-4-Second Autonomous Pipeline: Packaging complex multimodal vision analysis, market arbitrage calculation, and Firestore transaction logging into an ultra-fast serverless pipeline on Google Cloud Run.
Production-Grade Design & State Persistence: Building an enterprise-grade dark-mode GenUI in React with real-time persistent history, multi-currency conversion, and dynamic PDF waybill generation.

What we learned

This project fundamentally shifted how I view AI development. Moving beyond conversational chatbots into autonomous agentic workflows requires much more than just prompting an LLM. It requires defensive architecture; using specialized models like gemma-2-9b-it for security guardrails, leveraging gemini-3.6-flash for multimodal perception and tool execution, and anchoring everything with persistent transactional databases like Firestore.

Most importantly, I learned that great AI engineering in emerging markets isn't about adding more complexity; it's about stripping away user interface friction so that powerful cloud infrastructure works invisibly for the people who need it most.

What's next for KilimoAgent

Expanding Geographic RAG Calibration: Broadening the regulatory and trade corridor datasets beyond East Africa to include West and Southern African agricultural trading hubs (ECOWAS & SADC).
Live Bidirectional Video/Audio Streaming: Polishing the full-duplex Gemini Live API WebRTC integration to optimize performance under bandwidth-constrained rural 3G/4G networks.
Fintech & Instant Escrow Integration: Connecting mobile money payment rails (M-Pesa, Airtel Money) directly to the generated waybills so farmers receive automated milestone payouts upon freight pickup.
Direct IoT & Scale Integration: Hooking digital warehouse weighbridges directly into the agent intake pipeline to cross-verify physical grain weight against farmer voice declarations.