Smart home voice agent

An AI-powered smart home assistant using Whisper and Qwen LLM to turn natural speech into real-time MQTT hardware control via ESP32.

Comment

Inspiration

Traditional smart home assistants often rely on rigid, pre-programmed keywords. Saying "Turn on the light" works fine, but if you say "It's getting pretty dark in here," conventional interfaces often fail to parse the context.

We were inspired to build a system that bridges the gap between human intuition and IoT hardware. By combining local speech recognition with modern Large Language Models (LLMs), we wanted to create an agent that feels like a natural conversation partner capable of managing physical home environments without requiring users to memorize strict command structures.

What it does

  1. The Smart Home Voice Agent (SHVA) is an end-to-end voice automation ecosystem that translates natural speech into real-time physical device actions.
  2. Contextual Intent Recognition: Understands natural phrases like "It's getting too hot in here" to activate the fan, or "It's time for bed" to turn off all devices.
  3. Hardware Execution: Controls lighting, ceiling fans, televisions, and media playback via an ESP32 microcontroller using low-latency MQTT messaging.
  4. Environmental Sensing: Continuously monitors ambient indoor climate data using a DHT22 sensor and provides visual and voice feedback.
  5. Control Dashboard: Offers a web-based Flask interface featuring custom UI themes (Sunrise, Light, Dark), live activity terminal logging, and remote tunnel support.

How we built it

The system architecture combines edge hardware with modern AI workflows:

  1. Speech-to-Text Pipeline: The INMP441 audio microphone feeds into a local speech recognition pipeline powered by Whisper AI via faster-whisper for low-latency transcription.
  2. Intent Parsing Engine: Raw transcription text is passed to Qwen LLM via DashScope API. The prompt structure constrains the output into structured JSON intents (e.g., {"device": "FAN", "state": "ON"}).
  3. IoT Messaging Backbone: A Mosquitto MQTT Broker routes generated commands to an ESP32 module over lightweight pub/sub topics.
  4. Hardware Driving Layer: The ESP32 parses incoming MQTT payloads to switch relays, adjust PWM driver signals (for fans), read DHT22 metrics, and play audio feedback through a MAX98357 I2S amplifier.
  5. Dashboard & Web UI: Built with Flask, raw web requests interface with local hardware states over REST API endpoints, styled with CSS variable-based glassmorphism interfaces and realtime SSE logging.

Challenges we ran into

  1. Audio Streaming & Latency: Streaming raw PCM data from the ESP32's I2S microphone over local networks without dropping audio packets or causing high transcription latencies proved difficult. We resolved this by optimizing local audio buffer sizes and implementing efficient Whisper model quantization.
  2. LLM Output Determinism: Generative models naturally produce conversational responses, which can be difficult for embedded microcontrollers to parse. We engineered strict system prompts to force the LLM to output predictable JSON payloads while preserving contextual understanding.
  3. Hardware Power & Signal Integrity: Driving inductive loads like DC motors and relays alongside sensitive I2S audio components caused voltage drops and noise on the micro-controller power rails. Isolating circuit ground paths and adding decoupling capacitors stabilized audio clarity.

Accomplishments that we're proud of

  1. End-to-End Orchestration: Successfully linking human speech through cloud/local AI and back down to low-level microcontroller output with minimal latency.
  2. Robust Context Parsing: Building a system that accurately maps abstract user complaints ("I can't see my book") to exact physical actions (LIGHT_ON).
  3. Sleek Web Interface: Designing a fully custom Flask UI with live terminal logs, real-time volume controls, theme customization, and mobile-friendly layouts.

What we learned

  1. Edge-to-Cloud Integration: Balancing tasks between edge devices (ESP32 sensor polling) and local/cloud servers (Whisper transcription & LLM inferencing) is key to responsive IoT architecture.
  2. Prompt Engineering for Hardware: Prompt design is as crucial as firmware code when translating human speech into reliable hardware control vectors.
  3. MQTT Reliability: Implementing robust fallback states and state-synchronization loops between the web server, database, and microcontrollers prevents race conditions when controlling physical devices.

What's next for Smart home voice agent

  1. Edge AI Integration: Onloading lighter voice models directly to microcontrollers or local NPU edge devices (like the ESP32-S3 or Raspberry Pi 5) for 100% offline functionality.
  2. Multi-Room Support: Expanding room-aware context parsing using multi-node microphone arrays to determine which room the user is speaking from.
  3. Predictive Automation: Utilizing historical sensor data (temperature, time of day, routine preference) to automate appliance control proactively without requiring manual voice prompts.

Built With

Share this project:

Updates