Every security camera we'd ever used answered the same, flat question: "did something move?" That's it. No context, no understanding of who was there or why, no way to tell a delivery driver from an actual threat...
We wanted to build something that didn't just detect it would understand and respond. Instead of a motion alert buried in an app nobody checks until it's too late, what if the system could actually talk to whoever was standing at the door? Verify who they are by voice, ask what they want, and only escalate if something's genuinely wrong. That question "what if the camera could have a conversation instead of just ringing a bell?"
What it does
PancakeStack watches a camera feed and only pays attention when it actually matters:
- we only escalate once a meaningful fraction of the frame is actually moving, so a swaying tree branch doesn't wake the whole pipeline.
- YOLOv8 verifies that what's moving is actually a person, with a confidence score, not just "some pixels changed."
- voice authentication kicks in once a person is confirmed: the system listens, transcribes speech, and checks the visitor's voice embedding against authorized profiles.
- A security phrase acts as a second factor — voice match and passphrase match both have to be true before anything is treated as "authorized."
- Gemini reasons over all of that context (who's speaking, whether they're authorized, what they said) and drafts an appropriate, conversational response instead of a canned alert.
- ElevenLabs speaks that response out loud, so the interaction feels like talking to an actual security assistant, not a beep.
How we built it
We treated this as a pipeline of independent, swappable layers rather than one monolithic script — which turned out to be the single most important architectural decision we made (more on that below).
Vision : Python + OpenCV, with a background-subtraction motion gate in front of a YOLOv8 person detector, so the expensive model only runs when there's real motion to justify it.
Audio : SpeechRecognition for transcription and Resemblyzer for voice embeddings, compared against pre-enrolled authorized voice profiles.
Reasoning layer: Gemini, fed the transcript, the voice-match result, and the security-phrase result, to decide what to actually say back.
Voice output: ElevenLabs converts Gemini's text response into speech, played back through a speaker.
Config API keys and voice IDs live in a .env file rather than anywhere near source control.
Challenges we ran into
The Raspberry Pi camera fought us the entire way. Our original plan was a dedicated Pi running the whole vision pipeline. We hit driver conflicts, incompatible camera libraries, and OpenCV backends that either silently failed or handed us black frames instead of an error. The fix wasn't a clever workaround. we accepted that camera capture and AI processing didn't need to live on the same device, and decoupling them into their own layer. Once we stopped forcing everything onto one board, we could debug each piece independently instead of guessing which of five stacked systems was actually broken.
Camera access failures don't always look like errors. Even after moving off the Pi, we kept hitting the exact same failure mode on other hardware: cv2.VideoCapture reporting the camera as open while every frame read came back empty or black. The lesson we took away is that "camera opened successfully" and "camera is actually giving you frames" are two different guarantees, and OS-level permission systems will often fail silently rather than throw so a proper diagnostic script (checking multiple backends and printing actual pixel data, not just the isOpened() flag) saved us more time than any amount of guessing.
Python dependency hell A ModuleNotFoundError: pkg_resources mid-demo prep sent us back to rebuild the virtual environment from scratch and pin package versions properly instead of trusting whatever pip install happened to resolve.
The assistant talked over itself. Once voice input and voice output were both live, the system's own speech would occasionally get picked up by the microphone and treated as new visitor input — a feedback loop. We fixed it with an assistant_speaking flag that mutes the input pipeline while the system is talking.
API plumbing. Getting Gemini, ElevenLabs, and voice-ID keys all authenticating cleanly, without hardcoding anything, meant getting disciplined about environment variables earlier than we wanted to.
Log in or sign up for Devpost to join the conversation.