HomeSense

🏡✨ HomeSense uses AI to explain smart-home failures, safely undo unwanted changes, and test fixes before they happen again! 🔄💜

Comment

Inspiration

Smart homes are supposed to make life easier. But what happens when they make a mistake?

Imagine waking up at 2 AM because your hallway lights suddenly turned on, your living room lit up, and your thermostat raised the temperature—all while Night Mode was enabled.

You know something went wrong. But why did it happen? Which automation caused it? And how do you fix it without accidentally making things worse?

Most smart-home systems are designed to execute commands, not explain their decisions. When automations conflict, users are often left digging through settings, disabling routines, or manually restoring devices.

We wanted to change that.

Inspired by the growing capabilities of Alexa+, agentic AI, and the idea of human-centered automation, we built HomeSense — Your home, understood. 🏡💜

Our vision was simple: A truly intelligent home shouldn't just follow commands. It should explain what happened, help you safely undo mistakes, and let you test solutions before trusting them.

We also wanted intelligent home automation to feel approachable. Instead of another intimidating technical dashboard, we imagined a colorful miniature home with an adorable AI companion that helps you understand what's happening behind the scenes.

That's how HomeSense was born.

What it does

HomeSense is an AI-powered smart-home intelligence companion that makes automation explainable, reversible, and testable.

It combines two complementary capabilities: causal investigation and safe recovery.

When unexpected behavior occurs, HomeSense follows a five-stage workflow:

🔎 1. Understand what happened

HomeSense records simulated device events, automation evaluations, state changes, and causal relationships.

Users can ask natural questions such as:

"Why did my lights turn on at 2 AM?"

Using Groq-powered AI and real Model Context Protocol (MCP) tools, HomeSense investigates the event history and explains what happened using recorded evidence.

Rather than providing a generic guess, it identifies the triggering sensor, responsible automation, affected devices, and supporting event IDs.

🧩 2. Follow the causal chain

HomeSense turns technical event logs into an interactive visual explanation.

For example:

Hallway motion detected → Welcome Home automation triggered → Night Mode condition missing → Lights and thermostat changed.

Users can explore the causal graph, view the chronological event timeline, and expand the evidence behind the explanation.

🔄 3. Rewind unwanted changes safely

This is where HomeSense goes beyond traditional smart-home monitoring.

Users can selectively reverse unwanted automation effects.

If an incident changed two lights and a thermostat, users can choose to restore only the lights.

Every recovery proposal includes previous values, current values, proposed changes, and safety conditions.

Importantly, the AI cannot authorize its own actions. Device recovery requires explicit human approval, and the deterministic safety engine verifies device state and revision before execution.

Newer manual changes are protected from stale rollbacks, and sensitive security recordings cannot be deleted through recovery.

🛠️ 4. Repair the responsible automation

HomeSense can propose a minimal, targeted improvement to the automation that caused the problem.

For the Night Mode incident, it identifies that the Welcome Home routine lacks a condition preventing execution during nighttime.

It proposes adding:

AND Night Mode = disabled

Users can inspect the exact change before independently approving it.

🧪 5. Test the fix with Counterfactual Replay

Our signature feature is Counterfactual Replay.

Instead of applying a fix and hoping for the best, HomeSense recreates the original incident inside an isolated simulation using the updated automation rule.

Users can compare two miniature homes side by side:

  • What happened: Hallway light 100%, living-room light 80%, thermostat 25°C.
  • What would happen: Hallway light 10%, living-room light off, thermostat 21°C.

The replay also verifies that the repaired automation still works during normal daytime conditions.

The primary scenario runs ten verification assertions, testing the intended fix, device states, security preservation, and regression behavior.

All of this happens inside a playful, pastel-colored interface featuring an interactive dollhouse, expressive HomeSense companion, visual evidence cards, and a simulated Alexa+ conversational experience.

HomeSense is a software simulation, not a connection to production Alexa+ or physical smart-home devices. Its purpose is to demonstrate how explainable, human-controlled agentic automation could work.

How we built it

We designed HomeSense around one principle:

AI should help people reason about the home, but deterministic systems should remain responsible for executing and validating changes.

That separation shaped the architecture.

🧠 AI reasoning with Groq

We integrated Groq's openai/gpt-oss-20b with low reasoning effort to power natural-language investigation.

The agent interprets questions, chooses available tools, retrieves information, and generates evidence-grounded explanations.

It uses sequential tool calling, bounded execution loops, schema validation, provider timeouts, and rate-limit handling.

If Groq becomes unavailable, HomeSense transparently switches to a deterministic demonstration mode without misrepresenting the response as live AI.

🔌 Real MCP integration

For the Alexa+ track, we implemented a genuine Model Context Protocol server and client using the official MCP SDK.

The system uses MCP specification version 2025-11-25 over authenticated Streamable HTTP.

HomeSense exposes 14 discoverable tools supporting operations such as:

  • trace_incident
  • get_incident_evidence
  • get_device_state
  • get_automation_rules
  • propose_recovery_plan
  • apply_approved_recovery
  • propose_rule_patch
  • replay_incident

The application's agent uses the actual MCP client to investigate the simulated home.

We also built a Developer workspace that exposes tool invocation names, correlation IDs, execution status, and measured latency, making the integration inspectable rather than hidden behind the interface.

⚙️ Deterministic smart-home engine

The underlying simulation models six smart-home devices, including lights, a thermostat, sensors, and a security camera.

Structured automation rules execute through a deterministic evaluation engine with state revisions, validated conditions, priorities, and loop protection.

An append-only event journal records device changes and automation behavior, including causal relationships.

This provides a reliable evidence foundation for AI investigations.

🛡️ Human-controlled recovery

We developed a compensation-based recovery engine that checks current device states and revisions before reversing changes.

Recovery and automation repair use separate, expiring, single-use approvals.

The LLM can propose actions, but it cannot approve or independently execute them.

Security-sensitive actions are protected by explicit policy controls.

🏠 Frontend and experience

We built the application using React, TypeScript, Vite, and custom SVG/CSS illustrations.

The visual centerpiece is an interactive miniature dollhouse whose device states reflect the underlying simulator.

HomeSense also includes responsive conversations, an animated assistant companion, evidence exploration, human-approval dialogs, and a side-by-side replay theater.

The backend uses Node.js, Express, and SQLite, with isolated demo sessions and local event persistence.

Our application is organized into separate frontend, backend, and reusable core packages.

🧰 Tech stack

Frontend: React, TypeScript, Vite, SVG, CSS
Backend: Node.js, Express, TypeScript
AI: Groq, OpenAI GPT-OSS 20B
Agent tools: Official Model Context Protocol SDK, Streamable HTTP
Database: SQLite
Testing: Vitest, Playwright
Deployment target: Render
Version control: Git and GitHub

Challenges we ran into

1. Making AI explanations trustworthy

One of the hardest problems was ensuring that the assistant explained actual recorded events rather than confidently generating plausible but unsupported conclusions.

Early live testing revealed that the model could produce overly detailed responses and occasionally introduce unsupported contextual assumptions.

We strengthened the grounding process by validating evidence references, preserving MCP-derived metadata, and separating observed facts from inference.

Users can now expand an explanation to inspect its underlying evidence.

2. Groq rate limits during agentic workflows

Our live Groq integration successfully answered an unfamiliar question using the real trace_incident MCP tool.

However, a follow-up investigation encountered HTTP 429 after invoking another MCP tool.

Rather than pretending the entire conversation remained AI-generated, HomeSense correctly identified the provider limitation and switched to deterministic fallback.

We subsequently implemented bounded retry behavior, shared cooldown handling, and accurate per-message provider attribution.

A complete live Groq multi-turn sequence has not yet been independently verified, so we document that limitation transparently.

3. Building safe undo for changing device states

Reversing an automation sounds simple until the user changes a device manually after the original incident.

Blindly restoring an old value could overwrite a newer, intentional choice.

We solved this by checking expected state and revision numbers before compensation, blocking stale operations, and requiring explicit approval.

Our State Drift scenario demonstrates this behavior by preserving a newer manual thermostat setting.

4. Making counterfactual testing meaningful

We wanted more than an animation showing a successful fix.

The replay engine needed to operate on an isolated snapshot, rerun the original trigger, and verify actual resulting states.

It also needed regression tests proving that the corrected automation still functions under intended daytime conditions.

This became one of HomeSense's most technically substantial components.

5. Balancing technical depth with an approachable experience

Our first interface looked like an enterprise observability dashboard.

Although functional, it didn't communicate the warmth and simplicity we wanted for a consumer smart-home companion.

We redesigned it into a colorful, Alexa-inspired miniature world while preserving the technical evidence, interactive tools, and approval workflows.

The challenge was making complex causal reasoning feel intuitive without hiding important safety details.

6. Making the application reproducible

HomeSense combines a frontend, persistent backend, SQLite state, real MCP transport, and an optional external AI provider.

We encountered environment configuration issues, local networking restrictions during testing, and deployment-specific port-binding requirements.

For Render compatibility, we adapted the server to listen on 0.0.0.0 in the hosting environment while retaining loopback defaults for local development.

We also tested the public repository from a clean checkout rather than relying on dependencies and secrets already present on the development machine.

Accomplishments that we're proud of

🏆 A complete explain → undo → repair → verify workflow

Our biggest achievement is connecting all four capabilities into one functioning experience.

HomeSense doesn't stop after diagnosing an issue. It can propose safe compensating actions, obtain separate approvals, repair the relevant automation, and evaluate the proposed improvement.

🔌 Genuine MCP tool orchestration

We implemented an official-SDK MCP server and a working client rather than treating MCP as a label in the technology stack.

Its tool calls are observable, testable, and connected to actual simulation state.

🧪 Counterfactual Replay with ten verification assertions

We built an isolated replay system that evaluates both the original failure and the revised behavior, including daytime regression checks.

🛡️ Safety as an architectural property

The AI cannot approve itself, override security restrictions, or blindly restore stale device states.

We are particularly proud that the application can say "I don't have enough evidence" or "This action is blocked" instead of forcing a confident answer.

🎨 A distinctive miniature smart home

Our custom dollhouse and friendly HomeSense orb make the technical workflow visually accessible and memorable.

The same home can communicate an unexpected event, show the consequences, and illustrate the repaired outcome.

✅ Strong automated validation

Our deployment-readiness verification reported:

  • 59 automated tests passing
  • 16 browser tests passing
  • Successful typecheck, lint, and production build
  • Passing production smoke tests
  • Real MCP transport verification
  • Ten passing primary replay assertions
  • Frontend credential-exposure checks

These results were obtained during local verification, including a clean-checkout release test. We distinguish automated/mock-provider validation from live external AI validation.

Most importantly, we developed a coherent proof of concept showing how AI-assisted home automation could become more accountable and understandable.

What we learned

Building HomeSense taught us that the most important capability of an AI agent isn't always taking action. Sometimes, it's knowing when not to act.

We learned that reliable agentic applications need more than a powerful language model.

They need trustworthy evidence, structured tools, explicit permissions, state-aware execution, and mechanisms to verify outcomes.

We also learned that explainability becomes much more valuable when it connects directly to actionable recovery.

Understanding a problem is helpful. Understanding it, correcting it safely, and testing the correction is transformative.

Working with MCP demonstrated the value of standardized interfaces between AI reasoning and external capabilities. By keeping HomeSense's domain logic separate from its tools and model provider, we could test important behavior independently of live AI availability.

The Groq integration taught us to treat rate limits, partial failures, and uncertain outputs as normal engineering conditions—not unusual exceptions.

And redesigning the frontend reinforced another lesson: sophisticated engineering doesn't need to look intimidating. A delightful interface can make complex systems more understandable without compromising their integrity.

What's next for HomeSense

HomeSense is currently a simulated-home prototype, but the underlying idea has potential beyond the hackathon.

🏠 Real smart-home integrations

Explore supported device ecosystems and authorized APIs so HomeSense can investigate actual automation histories and device states, while maintaining strict privacy and permission boundaries.

🧠 Richer multi-agent investigation

Develop specialized investigation, recovery-planning, and verification agents, with independent evidence validation and clearly defined responsibilities.

🔁 More powerful counterfactual testing

Expand the replay engine to analyze multiple interacting automation rules, additional environmental conditions, and larger regression suites.

🗣️ Natural voice interaction

Bring the experience closer to the convenience of a voice-first smart-home companion, while preserving explicit approval for sensitive operations.

👨‍👩‍👧‍👦 Household-aware permissions

Introduce authenticated household identities, role-based permissions, shared-device ownership, and transparent approval histories.

🔐 Production-grade reliability

Add durable hosted storage, stronger user identity, distributed rate limiting, monitoring, and secure integrations before considering physical-device operation.

🌱 Open-source building blocks

Explore extracting reusable components for evidence-grounded agent workflows, safe compensating actions, and counterfactual testing.

Our long-term vision is a future where smart homes aren't merely automated—they're understandable, accountable, recoverable, and deserving of their users' trust.

HomeSense — Your home, understood. 🏡💜

Built With

Share this project:

Updates

Submission history