HomePilot

Your helper there for you
The tech stack
apis usd

Inspiration

What it does

HomePilot lets you add groceries and check your list using voice or typed commands. HomePilot also lets you hold a product up, to your webcam or upload a photo. HomePilot identifies products asks for confirmation before adding them and asks you to name unfamiliar items. HomePilots animated character brings the assistant to life. HomePilot feels like a friend.

How we built it

We created the interface using Next.js and React. We use access to the camera and microphone in the browser. ElevenLabs handles the transcription of speech. Our FastAPI backend manages the requests. Qwen3 works on our machine through Ollama to understand commands and pick the right grocery-list tools. SigLIP2 is loaded using Hugging Face Transformers. Compares product photos, with a reference gallery. Python MCP tools. Update a local SQLite database.

Challenges we ran into

Connecting speech, camera capture, product recognition and language-model reasoning required coordination. Recognition also had to handle products and the assistant had to decide whether the assistant should just suggest an addition or actually save the product.

Accomplishments that we're proud of

The implementation of so many different apis and machine learning in such a short time is an accomplishment. We learned of many new useful implementation of those apis. We also deepen our skills as coders and can celebrate getting a project across the finish line in such a short time.

What we learned

We learned how to connect AI models through APIs and give each component a clear responsibility. We also learned that reliable AI interactions depend on reference data, input validation and clear user feedback—not model predictions.

What's next for Home-assistant

We want to expand the product gallery, improve recognition across different lighting conditions and camera angles, and add household inventory tracking and history.