Why I made this
I did not just want an AI with a camera. I wanted an AI that could become interested.
Most multimodal assistants see only after a person starts a call, shares a camera, or asks a question. Shin explores something different: an embodied companion that can begin a conversation from either an inner motivation or a meaningful change in the shared physical world.
I think of it as not just autonomous AI, but spontaneous AI.
Two directions of spontaneity
Shin is built on Kota Mizushima's MIT-licensed open-source project familiar-ai.
The upstream project already included an opt-in Desire loop. It was disabled by default in my initial setup, so I had not tested it before Build Week. I have now enabled and validated it on my physical Tapo C220: with no voice or text prompt, an internal Desire started a turn, chose to pan and use the camera, interpreted the room, and spoke through the camera speaker.
That is inside-out initiative: the agent's internal state gives it a reason to act.
My Build Week contribution adds a separate path: Ambient Noticing.
That is outside-in initiative: a stable change in the environment becomes a reason to pay attention, judge whether speaking is worthwhile, and either begin a conversation or remain silent.
The working hardware demo
The core hardware run was recorded as one continuous 89-second test. The submitted 2:14 demo wraps that run with narration, English captions, and a brief privacy-preserving transition.
In the underlying hardware run, both paths operate in the same process, with speech recognition idle and no text prompt:
An upstream internal Desire starts a turn. Shin chooses the camera, pans, observes me holding a phone, and comments on the room aloud. I then start Ambient Noticing and show a pink plush toy. The local detector identifies a stable visual change. GPT-5.6 compares the scene before and after and decides that the deliberately presented object is worth responding to. Shin speaks once through the physical camera:
“The pink plush came closer. It looks soft—like the world got a little brighter.”
The responses look similar on the surface because both are unsolicited. Their causes are different: one begins inside the agent; the other begins in the world around it.
How Ambient Noticing works
A lightweight local detector samples the latest camera frame using downscaled grayscale differences. Exposure shifts are compensated for, and changes must remain stable across consecutive samples. Only a bounded before/after pair and normalized change signal are sent to GPT-5.6. A structured decision layer evaluates confidence, novelty, camera-directed action, relevance, user priority, protected context, and interruption value. Minor lighting changes, small posture shifts, ambiguous motion, and low-confidence interpretations fail closed. If accepted, exactly one generated response enters a guarded delivery path and is spoken through ElevenLabs and go2rtc to the Tapo camera speaker. GUI status exposes detection, semantic interpretation, the speak-or-stay-silent decision, response generation, and audio delivery.
This avoids continuously sending video to the model and makes silence part of the product design.
What existed before / what I built during Build Week
Upstream familiar-ai provided:
the embodied ReAct agent camera vision and PTZ tools speech input and output memory infrastructure cognitive components and internal Desires the opt-in Desire-driven idle loop
I added during Build Week:
continuous lightweight scene-change detection while Ambient Noticing is active GPT-5.6 before/after semantic interpretation an explicit interest and interruption-value decision confidence, cooldown, privacy, and user-priority gates environment-originated response delivery and physical speech GUI observability and durable provenance for ambient-originated messages focused regression tests and safe TTS/provider diagnostics
The contribution is not “making familiar-ai speak first.” It is connecting the agent's attention directly to events in the shared physical world.
The original MIT license and Kota Mizushima's copyright notice are preserved.
How I used Codex and GPT-5.6
I am a non-technical builder and creator. Codex was my implementation partner: it traced the existing asynchronous architecture, translated my behavioral requirements into bounded changes, wrote focused tests, and helped diagnose WSL, GUI, configuration, camera, semantic-policy, and TTS failures.
I made the product decisions, defined what “interest” should mean, corrected assumptions when behavior did not match the real device, and performed the physical-camera tests.
GPT-5.6 is part of the runtime product. It interprets the visual difference and makes the semantic decision: is this change clear, relevant, and valuable enough to interrupt for?
Why this is not video chat
Video chat gives an AI sight when a person deliberately opens a visual conversation.
Ambient Noticing gives an AI attention: the ability to notice a change without being asked, decide whether it matters, and choose between speaking and silence.
Sight asks: “What can the AI see?”
Attention asks: “What does the AI find worth responding to?”
Current limitations
The upstream internal-Desire path can surface episodic recollections that are not sufficiently grounded in verified history. In testing, it produced plausible but inaccurate references to “yesterday.” Reliable memory provenance is an important next step.
Ambient Noticing currently works best with a stationary camera and clear, stable scene changes. Its sensitivity and conversational judgment still need calibration for long-term everyday use.
The prototype now supports both directions:
mind → world: an internal drive causes Shin to look and act world → mind: an external event captures Shin's attention
Log in or sign up for Devpost to join the conversation.