Clutter Bot

Inspiration

Most people who live with clutter don't lack motivation. What stops them is deciding, for every single object, where it should go. Faced with that many small decisions, it's easier not to start, and "cleaning" turns into moving piles from one surface to another. We wanted something that makes those decisions for you, gives you one easy step at a time, and then actually checks that the job got done.

What it does

ClutterBot lives in iMessage. You text it a photo of your messy desk or room.

1. Observe: Perceptron finds every item and every place things can go (bins, drawers, cups), and ClutterBot sends back your photo with each item numbered.
2. Ask: It asks two or three quick questions a camera can't answer: what you use every day, what's trash, and which bin is already full.
3. Guide: It gives each item a home and a one-line reason, then walks you through the cleanup one step at a time, by text or by phone call (voice agent on Photon).
4. Check: You send an "after" photo, and Perceptron checks each item: verified (seen in its new spot), can't confirm (gone from the desk, but we can't see where it went), or still here.

ClutterBot never claims something is put away just because you said so. It keeps what the camera saw separate from what you told it.

How we built it

• Perceptron Mk1.5 /v1/detect with a fixed list of item names, so the before and after photos use the same labels. Storage spots are outlined using Perceptron's polygon annotations.
• Photon 2.0 for the iMessage conversation and the phone number for voice calls.
• Gemini turns the detections plus your answers into a structured plan, and generates a clearly labeled "what it could look like" preview.
• TypeScript / Node, sharp for drawing the numbered overlays, zod for validating every model response, and Vitest for tests.
• We wrote specs first (requirements, UML, test cases per feature) and shipped in small PRs reviewed by VEYZL.

Challenges

• The real API didn't match our assumptions. Detection responses leave out fields instead of sending null, which broke our response validation until we fixed it.
• The same object got counted twice. A notepad came back as both "book" and "notebook", earbuds came back as both "headphones" and "cable", and a pen cup was labeled "pen". We added cleanup rules: remove duplicate boxes, map synonyms to one name, and drop items that are really containers.
• Speed. One detection call takes about 12 to 13 seconds, and turning on reasoning pushed it to 38 seconds. So we keep reasoning off, run the item and storage-spot calls at the same time, and reply instantly with "looking…" so the user isn't left waiting in silence.

What we learned

• Open-ended detection produced sentence-long labels. A fixed label list was more accurate and made before/after comparisons possible.
• Measure latency on real photos early. Shrinking the image didn't speed things up at all, because the time was spent on the server.

What's next

Multi-room sessions, short video clips as proof that something went in a drawer, and a pilot in a residence hall.

Built With

• computer-vision
• elevenlabs
• gemini
• google-genai
• hono
• imessage
• node.js
• object-detection
• perceptron
• photon
• sharp
• sqlite
• typescript
• veyzl
• vitest
• voice-agent
• zod