Your home. Your intent. Your control.

Inspiration

Smart homes are supposed to make life easier, yet controlling multiple devices often means switching between apps, adjusting individual settings, and remembering specific commands. We wanted to change that experience by building a home that understands what people want rather than simply following isolated instructions. This inspired us to create Homepilot, an AI-powered smart home orchestration platform that turns everyday intentions into coordinated actions.

What it does

Homepilot allows users to control their smart home through natural language and voice commands. Instead of manually operating individual devices, users can express their goals, such as setting up a movie night, and let the AI orchestrate the required actions.

Key features include:

Goal-oriented AI orchestration: Converts natural language requests into actionable steps.
Unified smart home control: Manages lighting, climate, media playback, calendars, shopping lists, and device diagnostics.
Multi-provider AI support: Works with NVIDIA NIM, Amazon Bedrock, and locally running models through Ollama.
Real-time execution tracking: Displays agent progress, tool statuses, and execution metrics through a live dashboard.
Self-healing error recovery: Allows the agent to respond to tool failures and attempt recovery.
Interactive device simulators: Visualizes changes to smart lights, HVAC controls, and Fire TV playback.
Model Context Protocol (MCP) integration: Provides a structured interface between AI agents and available tools.

How we built it

We built Homepilot using a modular architecture that combines a web dashboard, an AI orchestration backend, and an MCP-based tool execution layer.

Frontend: Next.js 16, React, and Tailwind CSS for the interactive dashboard.
Backend: Fastify and TypeScript for the orchestration server and API endpoints.
AI integration: NVIDIA NIM, Amazon Bedrock, and Ollama through a unified provider interface.
Tool execution: MCP tools with structured schemas for interacting with home devices and services.
Database: PostgreSQL with Drizzle ORM for user preferences, device information, application data, and agent execution traces.
Real-time updates: Server-Sent Events (SSE) and Redis Pub/Sub, with an in-memory fallback for state propagation.
Authentication and monitoring: Alexa+ OAuth 2.0 token verification and structured AWS CloudWatch logging.
Open-source contribution: Extracted a reusable Streamable HTTP transport package for integrating MCP servers with Fastify or Express.

The system also includes a multi-turn agent loop that processes tool results, handles failures, and continues execution when recovery is possible.

Challenges we ran into

One of our biggest challenges was coordinating multiple AI providers behind a consistent interface while maintaining reliable tool execution. Different providers have different approaches to tool calling, so we needed a unified abstraction that could support them without tightly coupling the application to a single model.

Another challenge was managing asynchronous execution and keeping the dashboard synchronized with the backend. We addressed this through real-time event streaming and Redis-based state propagation, with a fallback mechanism for environments without Redis.

We also had to consider failures during multi-step tasks. Rather than treating every tool failure as the end of a request, we designed a recovery loop that feeds error information back to the AI agent so it can adjust its next action.

Finally, integrating MCP transport, authentication, persistent telemetry, and interactive device simulations required us to balance modularity, reliability, and a smooth user experience.

Accomplishments that we're proud of

Built a goal-oriented AI orchestration system rather than a simple command-response chatbot.
Integrated three interchangeable AI backends: NVIDIA NIM, Amazon Bedrock, and Ollama.
Implemented 12 structured MCP tools covering home control, media, calendars, shopping lists, diagnostics, and personalization.
Created a real-time dashboard that visualizes agent execution and simulated hardware state.
Implemented multi-turn tool execution with self-healing error recovery.
Developed an independent Streamable HTTP MCP transport package for reuse in other applications.
Prepared the platform for Alexa+ integration with an add-on manifest and OAuth 2.0 authentication support.

We're especially proud of bringing AI reasoning, structured tool execution, and real-time visualization together in one modular platform.

What we learned

Building Homepilot taught us that useful AI agents require much more than a capable language model. They need well-defined tools, structured interfaces, reliable state management, error handling, and visibility into what they are doing.

We also learned the importance of designing systems that are not tied to a single AI provider. A unified provider interface makes it easier to experiment with different models, including locally hosted options.

Finally, implementing real-time updates and recovery mechanisms reinforced how important observability and graceful failure handling are when an AI system performs multiple actions to accomplish a single goal.

What's next for Homepilot

Our next goal is to move Homepilot closer to a real-world, personalized smart home experience.

Real hardware integration: Connect supported smart lights, thermostats, and entertainment devices instead of relying solely on simulators.
Deeper Alexa+ integration: Advance the existing integration work toward a complete, tested end-to-end experience.
Personalized automation: Use user preferences and contextual information to make routines more relevant.
Proactive assistance: Explore context-aware suggestions for energy efficiency, daily routines, and home management.
Improved reliability: Expand automated testing, tool-level validation, and recovery strategies for complex workflows.
Privacy-first local AI: Strengthen offline operation through locally hosted models.
Open-source ecosystem: Develop the MCP transport package further to make integration easier for other developers.

Our long-term vision is to make Homepilot an intelligent home companion that understands your goals, coordinates your devices, and adapts to your lifestyle—with less manual control and more meaningful automation.