Orb Companion

Inspiration

Think about the last time you felt lonely. Now think about how hard it was to say it out loud. Typing it into a chat window can feel like filling out a form, and calling someone can feel like too much. We wanted to build something that makes the first step easier. Orb Companion doesn’t wait. It notices you walk up, glows, and starts a conversation. It isn’t a replacement for people. It’s meant to be a gentle first step toward talking to someone, whether that’s a friend, family, or a professional.

What it does

Orb Companion is a physical, voice-driven companion. It wakes when it notices someone approaching. It listens, reads your facial expression, and answers out loud. When you speak to it through your computer, it recognizes your emotions and assigns each one a light pattern, showing you that it’s listening. The app runs on your computer; its replies come from the Claude API.

How we built it

The orb: Orb Companion has a soft, continuous form with no sharp corners. It communicates through light instead of a screen. Five white LEDs show its mood as brightness patterns: all bright for happiness, a dim even glow for calm, only the bottom lit for sadness, and a slow breathing pulse for stress. A single light circles the ring while it thinks, and it fades out when it falls asleep. Hardware: An Arduino UNO with an ultrasonic sensor detects when someone approaches and drives the LEDs. The Arduino only senses and lights. Everything else runs on the connected computer, which sends it brightness levels over a single USB cable. The computer’s built-in microphone and camera handle listening and seeing. The software and AI: A Python app transcribes speech and reads facial expressions on the computer. For each reply, it calls the Claude API (Claude Opus 5.5) with what you said, your expression label, and one webcam snapshot. Claude returns an emotion and a short spoken reply, and our code maps the emotion to a light pattern. We built and tested Orb Companion with Claude Code as our development partner. The dashboard: A live SpacetimeDB dashboard shows Orb Companion’s mood in color, along with its status and a timeline of events. Privacy and security: Orb Companion doesn’t save audio, video or transcripts to disk: sound and camera frames stay in memory, speech is turned into text on the Mac, and the dashboard database runs in memory only. Orb Companion clears its conversation memory every time it falls asleep. Replies are generated by the Claude API, so for each reply Anthropic receives the conversation text (up to the last 20 turns), the time and distance reading, the facial-expression label and one webcam snapshot; how long that data is kept is set by Anthropic’s API terms, not by Orb Companion.

Challenges we ran into

Hardware: Our biggest obstacle was the enclosure. 3D printers were in high demand, and we couldn’t get translucent material, which would have made Orb Companion’s glow livelier and more modern. Then one part printed incorrectly, and we ended up redoing most of the printing from scratch. We also took detours to track down missing parts, and the solid pieces didn’t arrive until the last moments. That left no time to glue everything together or print the top of the orb, so our demo video doesn’t show the complete prototype. Only one of us had hands-on hardware experience, so all the assembly fell to that person. Circuit problems and a spotty USB connection cost us hours more. In the end we had no time to add more sensors, including one that would let Orb Companion feel when it’s picked up. Software: We had no dedicated backend teammate, so one engineer stepped in to build the software side, including the AI integration, speech, and dashboard. We also hit occasional bugs along the way. Facial expression readings are noisy, and we ran out of time to add more moods, like “tired.” We also planned for Orb Companion to understand American Sign Language (ASL) but didn’t have time to build it.

Accomplishments that we're proud of

Despite those setbacks, we built a working prototype that combines physical hardware and conversational AI, not just a screen demo. Walk up and Orb Companion wakes, listens, answers out loud, and shows its mood in light. Several of us were new to hackathons, yet we combined software and hardware to bring AI outside the screen, and we delivered a working demo video. We designed and built an original hand-crafted base, and we reprinted most of the enclosure when the first print failed. We also built Orb Companion with privacy by design. Nothing is saved to disk, and conversations are wiped when Orb Companion sleeps. Most of all, we’re proud of building something that aims to bring a little life and company to people who feel alone.

What we learned

We started the weekend with big ideas, and our first lesson was how easy it is to overshoot and run out of time to build the core of the project. A group can also drift off track halfway through, and people end up with different interpretations of the same idea. We learned to rely on each other’s strengths and responsibilities to keep a good workflow and finish something. Stepping outside our usual roles made us better at all of it, and Claude Code was our development partner throughout. We also learned a lot about the thing we were building. Hardware and AI break in different ways, and each can hide the other’s bugs, so we had to debug both at once. Most of our worst bugs only showed up on the real board. Designing for approachability turned out to shape everything, from Orb Companion’s form and lights to the way it talks, and these choices matter as much as what the AI says. Facial expression readings are noisy, so Orb Companion treats them as a clue and weighs your words above your face. And because we were building for people who feel lonely, we learned to be careful about privacy and about not replacing real human connection.

What's next for Orb Companion

Standalone orb: All the circuitry inside the orb, with no computer needed. Built-in speaker and camera. Translucent material for glow, and color LEDs. Personalization: Recognition of different users, and personalization for each of them. Sensors: a motion (IMU) or tilt sensor so Orb Companion knows when it’s picked up or held, and a temperature sensor to feel a warm hand. Training: Better facial expression training to catch subtle change beyond happy and sad. ASL: Sign language detection and interpretation for users who are Deaf or hard of hearing. Orb Companion will sign back. Languages: Multi-language support. Tutor Mode: Teaching and quizzing you subjects like Python or calculus. Multi-Orb: Link several Orbs together for a superior combined experience.