Monika AI — Multi-Modal Web & Hardware Assistant

Monika AI bridges intelligent web backends with physical IoT automation, leveraging Google Gemini to execute complex real-time workflows across software and physical devices.

Comment

Inspiration

Traditional voice assistants and software automation agents operate within rigid digital boundaries. They can process natural language or execute localized API calls, but they lack physical context, edge-level execution control, and multi-modal sensory feedback. Monika AI was designed to bridge this divide—connecting high-level multi-modal intelligence with edge hardware microcontrollers for real-time automation.

Our goal was to architect a unified, production-ready system powered by Google Gemini that seamlessly bridges high-concurrency web management, real-time WebSocket communication, and low-latency physical hardware actuation.

Architecture & How We Built It

Monika AI is designed as a multi-tier, event-driven ecosystem optimized for low-latency streaming, deterministic state management, and high resilience:

  • Intelligence Core (Google Gemini API): Serves as the central multi-modal agent, parsing complex natural language, visual context, and audio streams into deterministic function calls.
  • Backend Framework (Node.js & Java Spring Boot): Manages stateful user sessions, authentication, and security validation, while routing high-frequency event streams over persistent WebSockets.
  • Hardware Edge (ESP32 / Embedded C/C++): Microcontroller firmware executing hardware actuation (relays, sensors, motors) and streaming real-time telemetry back to the cloud via encrypted socket pipes.
  • Frontend Operations Dashboard: High-performance, responsive UI providing real-time telemetry visualization, interactive console streams, and manual hardware overrides.

Key Technical Challenges

  1. Deterministic Hardware Function Routing: Large Language Models can produce non-deterministic schema variations. Interfacing directly with physical hardware requires absolute precision. We implemented strict JSON Schema validation layers between Gemini function calls and binary micro-controller commands to eliminate execution failure risks.
  2. Bi-Directional State Synchronization: Maintaining sub-second synchronization between web clients, central API services, and physical edge devices across fluctuating network conditions required designing lightweight WebSocket event queues with state-verification loops.
  3. End-to-End Latency Minimization: Optimizing the entire execution chain—from multi-modal prompt ingestion to edge relay switching—to ensure natural, sub-second physical responses.

What We Learned

Building Monika AI highlighted the necessity of rigid schema enforcement when linking generative AI to physical hardware. We gained valuable experience in optimizing Node.js and Spring Boot thread pools for high-concurrency WebSocket channels, managing edge memory constraints in C++, and structuring reliable multi-agent system workflows.

Mathematical Formulation

To ensure deterministic tool selection, our multi-modal decision engine routes inputs to the optimal action $T^*$ from available tools $\mathcal{T}$ based on context vector $C$ by maximizing the log-likelihood over Gemini's projected response distribution:

$$T^* = \arg\max_{T \in \mathcal{T}} P(T \mid C; \Theta_{\text{Gemini}})$$

To filter noise out of edge sensor streams (such as telemetry coming from the ESP32), signal attenuation $S(t)$ is calculated using an Exponentially Weighted Moving Average (EWMA):

$$S(t) = \alpha \cdot Y(t) + (1 - \alpha) \cdot S(t-1)$$

where $\alpha \in (0, 1)$ is the smoothing coefficient and $Y(t)$ represents the raw incoming telemetry reading.

Built With

Share this project:

Updates