Sara Smartest Speaker

Sara is the first smart speaker AI Agent. She can listen and talk and call a powerful hermes-agent with several skills, tools, scripts to do things impossible with current smart speakers.

Comment

Inspiration

I create Generative AI Art Frames, with 33" LCD or 13" e-ink Screens, see link. They are controlled by voice (whisper-1), use gptp-image-2 and kling 3 to generate content and ElevenLabs to speak. Then I started a project to orchestrate and control a group of AI Smart Frames for exhibitions. I installed hermes-agent on a Raspberry Pi 5 and I created Zoetrope-Admin skills. Then I added more skills and several gateways (telephony, email, voice, iMessage, telegram, bee.computer streaming voice instructions, ...). But I needed a better interface than the web, a chat app of a terminal, so I added a sara-kit board that has 3 microphones and amp for speakers, that I fitted with a Raspberry CM4.

What it does

I can wake my smarter speaker (codename "Sara") using "OK Sara" or "OK Daemon" using some wake words (using porcupine from picovoice or VOSK and some clever tricks). Both "personalities" can listen and talk in full duplex using GPT-realtime-2. The device hears across the room using it's 3 microphones. Sara can reply using GPT-5.5 and ask Hermes some help when she need to recall long term memory or delegate an action. Dameon is a bit dumber in it's conversation but he can execute any task using my hermes-agent setup with a lot of (homemade) skills, plugins, scripts.

For example I can ask :

  • "Please call my wife to tell her I will be an hour late.". And it will use it's twilio number and some twiML to call her and leave the message.".
  • "Create a portrait of Charles in my Hallways."
  • "Check the post-sales evaluations testimonials from last month, list top 10 recurring themes, highlight top 3 insights and publish a report on the web. Send me the URL by Telegram."
  • "Create the animals of the jungles on my 9 Zoetrope in various styles."
  • "When was the front door opened this morning ? Is there someone working in the kitchen ?".
  • "Create a newspaper with the latest news of last night football match with forecast for the next match and display it on the Ink Zoetrope. "

How we built it

It start with a baseboard that has microphones, amp for speakers and a slot for a CM4. I installed Debian Trixie and hermes-agent with a selection of CLI (codex, bee, himalaya, openhue, shelly-cli, meross-cli, switchbot, ...) and small servers (coded by Codex) for listening to bee.computer and viewing live with Looki L1 devices. Hermes Gateway integrates with telegram, mail, whatsapp, twilio SMS, telephony, ssh, switchbot, shelly, meross, openhue, honcho for long term memory, ...

Using Codex with GPT-5.5/GPT-5.6 I built a system wide service that manage the voice agent full duplex interactions and it integrate with hermes-agent using tool calls to it's API. I added a queue system to be able to ask hermes-agent several task in parallel. I manage a local memory system to allow continuous conversations with Daemon (hermes-agent) to keep context. While Dameon is executing, I can continue to talk with him and ask more requests or just informations from the web, from memory, etc. The device expose a local web interface to activate listening, read logs of conversations, view detailed requests and results with hermes-agent, manage a contact list, choose voices for Sara and Daemon.

Challenges we ran into

First Codex had to research documentation on the audio chip (ZL38063) features to enable full duplex using AEC (Audio Echo Cancellation). Then I built things progressively and each step had it's challenges. Integrating each tool was a mini project with their bugs and elliptic documentations. Getting the fine timing parameters of full duplex voice interaction right, managing the interactions between Sara and Dameon, each with their own voices when the streaming model (gpt-realtime-2) can only pick one, delegate to GPT-5.5 or Hermes when the streaming model was not up to the level required, managing parallel requests that generate responses that overlap, ...

Accomplishments that we're proud of

Building the best UX for the a very smart voice agent able to do powerful actions using a very cheap hardware combined with OpenAI SOTA models. We beat Google, Alexa and obviously Apple in excellence of the UX, available features now, flexibility (I can run one, two or many more agents if needed), potentially unlimited evolution (hermes-agent + GPT-5.6 can go very far) and costs.

What we learned

How to use Codex on the device to code litteraly itself, test and iterate at lightning speed. For example the device has 2 speakers for stereo. Codex used one to simulate Sara & Daemon Talking and the second to emulate the user speaking (using ElevenLabs API) while recording signals on all 3 microphones to find the best parameters. That was a ahah moment when I realised that AI can code device that run itself to evolve it's capabilities.

What's next for Sara Smartest Speaker

I want to create a 3D printed enclosure then created my own custom board with only what is required in the cheapest/smallest format possible, add a small UPS battery to have better shutdown when requests are running and evolve it's capacities by building more on hermes-agent. Ideally I would like to move to GPT-Live-1 when available in the API.

Results

This project won 3rd place at the OpenAI local Hackathon on July 19 2026. link

Built With

  • codex
  • gpt-5.6
  • gpt-realitme-2
  • hermes-agent
  • n
+ 5 more
Share this project:

Updates