Home Hero

An AI-powered home assistant that predicts, prevents, and fixes maintenance issues before they become costly problems.

Comment

Inspiration

Maintaining a home is something almost everyone deals with, yet the process is still surprisingly outdated. Most people wait until something breaks, then scramble through manuals, search forums, or pay for expensive emergency repairs. We were inspired by the idea that home maintenance should be proactive, simple, and guided by AI.

We also noticed that recent advances in computer vision and local language models made it possible to build something more useful than a basic chatbot. Instead of just answering questions, we wanted to create a system that could spatially understand appliances, guide users through maintenance, and help verify repair progress in real time.

Our goal became clear: build an AI assistant that helps people take better care of their homes while reducing stress, cost, and uncertainty.

What it does

HomeHero is an AI-powered home maintenance assistant that supports both proactive maintenance and reactive repair guidance.

Users can upload a photo of an appliance, and the app identifies it using Google Cloud Vision API. Once identified, HomeHero creates a maintenance profile with recurring tasks such as filter changes, seasonal servicing, cleaning cycles, and inspection reminders.

When something breaks, users can describe the issue or upload another photo, and HomeHero enters repair mode. It generates a structured step-by-step workflow tailored to the appliance and problem.

Instead of dumping a long block of text, the app guides users one step at a time:

  1. Explains the current action
  2. Provides safety warnings
  3. Offers optional voice instructions using ElevenLabs
  4. Waits for user confirmation or progress photos

At each stage, users can upload photos of their work. HomeHero uses Google Cloud Vision API to visually analyze progress and determine whether the correct panel was opened, the right component was found, or the step was completed properly.

If something appears incorrect, the system provides corrective guidance before advancing.

This creates a repair assistant that can both teach and verify, giving users confidence to solve problems themselves.

How we built it

We designed HomeHero as a responsive web application for fast demo iteration and reliability.

Frontend

We used ReactNative + Expo to create a cross-platform app that runs useing Expo Go.

Backend

We structured the backend around modular services:

  • Authentication
  • Appliance management
  • Room organization
  • Scheduling
  • Repair sessions

We used Prisma + PostgreSQL to store users, appliances, tasks, repair sessions, and history.

AI Layer

We used a hybrid AI system:

  • Google Gemma hosted on Vultr for local reasoning, user guidance and workflow creation
  • Google Cloud Vision API for appliance model recognition and maintenance progress verification
  • ElevenLabs for asking questions and receiving spoken instructions

Repair Engine

One of our most important technical decisions was using a state-machine workflow instead of freeform chatbot outputs.

Each repair session is broken into structured steps:

S_1 ----> S_2 ----> S_3 ----> ..... S_n

Where each state contains:

  • instruction
  • expected visual result
  • help prompts
  • next valid transitions

This made the system safer, more predictable, and easier to debug.

Challenges we faced

1. Balancing intelligence with reliability

We did not want users receiving hallucinated repair advice. Freeform AI responses can sound convincing while being wrong.

To solve this, we separated planning from execution and used structured workflows instead of unconstrained chat.

2. Image recognition in real-world environments

Appliances vary heavily in:

  • lighting
  • camera angles
  • branding
  • cluttered surroundings

We had to design the system so image understanding was helpful without depending on perfect detection. We did so by combining AIs so if vision fell through, LLMs could pick up the slack.

3. Scope management

Our original vision included mobile apps, AR overlays, smart home integrations, and advanced diagnostics.

For a hackathon, we learned to focus on the highest-value experience:

  • identify appliance
  • generate maintenance plan
  • guide repair interactively

4. Integrating multiple AI services

Combining local models, vision APIs, speech synthesis, scheduling systems, and frontend UX into one seamless product required careful architecture decisions. Using vision API before LLM allowed us to reduce tokens + speed up product recognition

Accomplishments that we're proud of

We are proud that HomeHero became more than a concept. It evolved into a complete product system combining:

  • computer vision
  • local AI models
  • scheduling automation
  • modern app UX
  • AI pipeline that doesn't break down in the middle

We are especially proud of:

  • Running Google Gemma locally on Vultr
  • Creating interactive repair flows that verify user progress
  • Designing a clean demo-ready interface
  • Turning complex AI tools into a practical consumer product

What we learned

We learned that successful AI products are rarely about just having the biggest model. They are about designing systems where models, workflows, and user experience reinforce each other.

We also learned that structured AI can outperform generic chat in practical settings. Users do not always need paragraphs of intelligence, they need the next correct step.

In a way, product usefulness can be thought of as:

[ \text{Value} = \text{Model Capability} \times \text{Workflow Design} \times \text{User Trust} ]

Even a powerful model creates little value if the workflow is confusing or unreliable.

Finally, we learned that simplifying scope early is one of the most important hackathon skills.

What's next for HomeHero

We see HomeHero becoming a long-term platform for smarter home ownership.

Next steps include:

  • AR-guided repairs with live overlays
  • Warranty and receipt tracking
  • Predictive maintenance using usage history
  • Smart sensor integrations
  • Marketplace links for replacement parts
  • Fine-tuned appliance diagnostic models

Our long-term vision is simple:

Make every home easier, cheaper, and less stressful to maintain through AI.

Built With

Share this project:

Updates