AURA - Context-Aware Smart Home Agent

An Alexa+-style smart home agent powered by Gemini function calling, authoritative state verification, human security challenge safeguards, and real-time execution event inspection.

Comment

Inspiration

As ambient voice assistants transition toward proactive, autonomous agents like Alexa+, bridging the gap between natural language understanding and deterministic, safe physical-world actuation is critical. While large language models excel at interpreting human intent, real-world execution requires strict boundary enforcement, authoritative state verification, and safeguards against unintended physical mutations. I built AURA to model this next generation: an agent that parses compound conversational intent through modern function calling while maintaining a verifiable execution pipeline and human authorization checkpoints.

What It Does

AURA simulates an advanced context-aware smart home agent with:

  • Compound Intent Orchestration: Interprets multi-action commands in a single prompt (e.g., "Dim living room light to 30%, set thermostat to 26°C, and lock front door") and executes each tool accurately.
  • Authoritative State Enforcement: All device mutations are validated against strict physical bounds (temperature 16–30°C, brightness 0–100%) and held in an authoritative server-side state manager rather than optimistic client state.
  • Physical Perimeter Security Challenge: Critical actions (such as unlocking the front door) automatically halt execution and issue a short-lived (60s), single-use cryptographic challenge token (sec_cf_...) requiring human confirmation.
  • Live Auditable Execution Inspector: Streams transparent backend events (user_request, intent_interpretation, tool_call, tool_result, state_verification) with expandable JSON arguments and zero fabricated telemetry.
  • Automated Scene Routines & Voice Control: Executes multi-appliance scenes (Leaving Home, Relaxing Evening, Good Morning, Secure Home) and supports hands-free speech input via the Web Speech API.

How I Built It

  • Google AI Studio: Engineered, prototyped, and tested the agent's function-calling schema, system instructions, and tool definitions directly within Google AI Studio.
  • Agent Orchestration & Function Calling: Utilized the Google Gen AI SDK powered by gemini-3.8-flash with strictly defined tool declarations (get_home_status, update_lighting, set_thermostat, set_door_lock, run_home_routine).
  • Frontend & UI: Developed an interactive dashboard with React 19, TypeScript, Vite, and Tailwind CSS, integrating the Web Speech API for voice commands.
  • Dual Architecture & State Engine: Built a fullstack Node/Express server with Vite middleware alongside a standalone Python Flask API with an automated test suite.
  • Resilient Fallback Engine: Features a deterministic local command parser that guarantees 100% operational functionality and security checks even without external network access.

Challenges I Ran Into

  • Multi-Turn Tool Loops: Designing an orchestrator loop that reliably handles multi-device dispatch, validates arguments, feeds execution outputs back into the model context, and caps recursive rounds safely.
  • Security Confirmation Invariants: Implementing single-use challenge tokens that automatically invalidate upon first consumption or timeout to prevent replay attacks on physical perimeter locks.
  • Authoritative Telemetry: Ensuring the execution inspector displays only genuine backend state transitions and parameter validation passes without synthetic telemetry.

What I Learned & What's Next

I demonstrated that pairing generative language models with strictly bounded execution tools and human-in-the-loop security challenges creates an interface that is both intuitive and production-safe. My next steps include expanding support for dynamic scheduled routines, multi-room sensor streams, and edge-device hardware bridges.

Built With

Share this project:

Updates

Submission history