Inspiration

Omni is a handheld assistant that builds any app you ask for.

A voice assistant answers and the answer is gone. You cannot tap it, change it, or come back to it later.

We wanted the opposite: say what you need, and a small app stays on a device in your hand. A list, a scoreboard, a tracker. Touch it, or speak again.

The first try was a smaller e-paper board. The screen could not keep up, so the finished Omni is a Raspberry Pi and an LCD.

What it does

Hold the bar and ask for an app. Omni builds it on the glass.

Generate. "Keep score for ping pong." A scoreboard appears, and you can tap it.
Change. "Three points for Logan." That same app updates.
Navigate. "Open settings." The page opens.
Answer. Ask a question, and Omni says the reply through the speaker.

Speech is recognized on the device. The app stays after you let go of the bar.

Voice, the app wheel, and a generated app

How we built it

The board

Raspberry Pi 3 Model B, a 5" 800×480 touch LCD, an I2S microphone, and a speaker. The picture is HDMI. Touch is the XPT2046 on SPI. The case is a top and bottom cover around that stack.

The microphone already uses the Pi's I2S bus, so the speaker could not be a second I2S amp. It is a small transistor driver on a PWM pin, with a diode across the speaker coil.

Raspberry Pi, display, microphone, and speaker

System architecture

Vosk turns speech into text on the device. Omni firmware sends that text and the contract to Gemini, and one structured turn comes back. ElevenLabs speaks the line. The simulator runs that same firmware before a feature ships.

System architecture

Speech

Speech in stays on the Pi. Vosk listens through the microphone and turns it into text on the device, so the words are ready before anything is sent to Gemini.
Speech out is ElevenLabs. The short reply comes back as audio and plays through the speaker.
Apps get pictures the same way they get controls. Gemini names an icon from an offline library of about 1,900, and the firmware draws it. A name that is not in the library is rejected.

AI contract

A language model answers differently every time. The contract makes every reply a fixed shape, so the firmware follows the operation instead of the wording.

{
  "op": "mutate",
  "say": "Point for Logan.",
  "actions": [{ "action": "increment", "target": "logan", "amount": 1 }]
}

The sentence can change. The operation is what runs.

AI reasoning

Gemini decides what was asked before it answers. The same words can mean a new app, a change to one already on the device, a page to open, or a question.

How Omni classifies a request

Simulator

The laptop runs the same firmware as the Pi. A feature is built and tried there before it ships to the device. Tests cover the firmware, speech, generated apps, speaker audio, and the contract. A bad reply is rejected instead of saved as an app.

The simulator

Challenges we ran into

The Pi has one I2S bus, and the microphone needs it. A normal I2S speaker board would have fought the mic for those pins. The speaker ended up on PWM, through a transistor, with a diode so the coil does not kick the pin when the signal turns off.
The first Omni was a Seeed XIAO ESP32-S3 and an e-paper panel. Partial refresh was still too slow for an app to feel real-time, so the display moved to the LCD.
Early replies from the model were just sentences. The device could not tell a new app from a question. The contract is what made a reply something the firmware can run.

Accomplishments that we're proud of

A spoken request becomes an app you can touch, on a board we wired and cased ourselves.
Vosk turns speech into text on the Pi, on the device. ElevenLabs speaks the reply back through a speaker we drove from a GPIO pin.
The simulator and the device are the same firmware. The test suite checks that firmware, and a malformed Gemini reply never becomes an app.

What we learned

A model is more useful here when it has to pick an operation than when it is free to write the interface.
One I2S bus means the microphone and the speaker cannot be the same kind of part. The speaker had to be a small analog circuit.
An e-paper panel is a calm display and a slow one. The apps we wanted needed the LCD.

What's next for Omni

This Omni works. The next one should be smaller.

The first board was a Seeed XIAO ESP32-S3 and an e-paper panel, and that size is still the one we want. The next pass is that form again, with an e-ink panel that can partial-refresh fast enough for the apps.
The contract can grow too. More modules for Gemini to build from means more kinds of apps from the same spoken request.
It also needs a battery. We never had a cell small enough to close into the case and still run the stack. A pack that fits, and can carry the device, finishes the handheld.