Uma

Uma is an autonomous, voice-enabled AI operations layer and digital twin that monitors, diagnoses, and optimizes physical hydroponic vertical farms in real-time.

Comment

Meet Uma: The Autonomous AI Brain for Physical Vertical Agriculture

Inspiration

Agriculture is facing a critical inflection point. As climate change destabilizes traditional farming, vertical hydroponic farming offers a path toward hyper-local, climate-resilient food production. However, indoor vertical farms are delicate, highly coupled biological systems. A small shift in water temperature can trigger a chain reaction that ruins an entire harvest:

\( \Delta T > 6.0^\circ\text{C} \implies \text{Dissolved } O_2 \downarrow \implies \text{Nutrient Lockout} \implies pH \text{ Drift} \implies \text{Crop Death} \)

Managing these systems currently requires constant human vigilance, manual calculations, and rigid, hardcoded automation. We wanted to build something different: an autonomous, cognitive operations layer that acts as the autonomic nervous system of the farm. We were inspired to bridge the gap between "Vibe Coding"—using rapid, natural-language-driven development—and "Physical AI"—connecting digital intelligence to real-world sensors, fluid dynamics, and biological feedback loops.

What it does

Uma is an autonomous AI agricultural intelligence layer and interactive Digital Twin designed to manage vertical hydroponic farms.

  • The Interactive Digital Twin & Builder: Emulates a vertical farm housing crops with highly distinct microclimate tolerances (e.g. Butterhead Lettuce, Genovese Basil, Cherry Tomatoes, Tuscan Kale, Strawberries). It features a custom drag-and-drop structural editor allowing operators to map custom sensor networks, water manifolds, pumps, and solenoids to represent their unique physical layouts.
  • Autonomic Dosing & Actuator Control: Uma monitors physical variables in real-time, operating under strict operational boundary constraints: $$ \text{Equilibrium Targets: } \begin{cases} pH = 6.0 & \( 5.5 \le pH \le 6.5 \) \ EC = 1.8 \text{ mS/cm} & \( 1.2 \le EC \le 2.5 \) \ T_{water} = 24.0^\circ\text{C} & \( 20 \le T \le 28 \) \ RH = 68\% & \( 55\% \le RH \le 75\% \) \end{cases} $$ Using multi-step GraphQL tool reasoning, Uma autonomously corrects perturbations by triggering physical dosing pumps, solenoids, or HVAC units.
  • Cascade Failure Simulation: Implements mathematical model propagation. For example, if temperature exceeds critical limits, it triggers a cascade calculation: $$ \text{If } T > 30.0^\circ\text{C} \implies \Delta DO_2 = -1.5 \text{ mg/L} \quad \text{and} \quad \Delta Q_{\text{flow}} = -0.2 \text{ L/min} $$
  • Hands-Free Bidirectional Voice Agent: Utilizing high-speed WebSocket audio streaming patterns (Azure OpenAI Realtime WebRTC), operators can speak naturally with the facility. Uma uses Voice Activity Detection (VAD) and text-to-speech to discuss telemetry trends, diagnose root rot or mold, and execute commands hands-free while working on the farm floor.
  • Human-in-the-Loop Orchestration: When physical labor is required (e.g., foliar treatment for Botrytis mold or manual root inspections), Uma selects the best-fit agronomist or technician from the active farm roster, drafts a detailed diagnosis, and dispatches a high-fidelity work order email via AWS SES with direct deep-links to their mobile worker terminal.

How we built it

We designed a modern, full-stack reactive architecture designed for zero-latency telemetry feedback:

  1. Visual Twin Interface: Built with a Next.js 16 (App Router), React 19, and TypeScript frontend, using Vanilla CSS styling variables to achieve a premium, glassmorphic "Afterhours-tier" dark theme.
  2. Autonomous Core: Implemented a custom multi-step LLM reasoning loop. We pass the complete digital twin configuration schema dynamically to Uma’s system prompt, allowing her to understand any custom-built farm topology instantly.
  3. Real-Time Voice Layer: Integrated an Azure OpenAI Realtime WebSocket interface to enable low-latency, bidirectional audio streaming, converting raw PCM16 voice streams to Float32 audio buffers dynamically.
  4. Database & Query Layer: Configured a MongoDB Atlas database running Mongoose models, exposed to Uma’s tools via a unified GraphQL Yoga schema for clean query execution.
  5. Notifications & Dispatch: Integrated AWS SDK Client SES to send transactional emails to technicians with embedded deep links to close out physical work orders.

Challenges we faced

  • Low-Latency Audio Pipelines: Managing base64-encoded audio packets over WebSockets, converting them to PCM16, and downsampling to a stable \( 24\text{kHz} \) audio buffer in the browser was challenging. We overcame this by writing custom Float32 array conversions and time-scheduling buffer nodes to avoid audio cracking.
  • Multi-Step Parallel Tool Execution: Making sure that Uma could run multiple diagnostics (e.g. reading 3 different row sensors) in parallel and execute actuator commands without hitting API limits or causing race conditions. We designed a robust parallel Promise.all execution layer inside the GraphQL tool loop.
  • Schema Portability: Translating complex physical topologies (pipes, nodes, rows, sensors) into a lightweight JSON structure (FarmSchema) that the AI could parse natively in its system prompt without consuming excessive token overhead.

What we learned

  • The Power of vision-assisted Telemetry: Feeding a live baseline schematic snapshot directly into the visual context of the AI alongside raw numeric data drastically increases its spatial diagnostic accuracy.
  • Autonomic Correction Precedence: Centralized chemical adjustments (dosing pH/EC at the main reservoir) are highly superior to localized solenoid closures, which can starve roots of dissolved oxygen. We codified this hierarchical logic into Uma's operational core.
  • Vibe Coding as an Accelerant: Working hand-in-hand with AI coding assistants allowed us to shift focus from mundane boilerplate code to deep architectural integration—focusing on fluid dynamics, real-time voice streaming, and hardware-to-software loops.

What's next for Uma

  • Physical Microcontroller Bridge: Hooking up physical ESP32 relays to the GraphQL API to control real-world peristaltic dosing pumps based on Uma's digital actuator outputs.
  • Computer Vision Pathology: Integrating camera feeds to let Uma run computer vision on crops, diagnosing leaf necrosis or calcium tipburn visual indicators before they show up in chemical telemetry readings.

Built With

  • aws-client-ses
  • azure-openai-api
  • graphql
  • graphql-yoga
  • lucide
  • mongodb-atlas
  • mongoose
  • next.js-16
  • react-19
  • typescript
  • vanilla-css
Share this project:

Updates