OpenCrawl
An offline voice agent that grows new abilities when you plug hardware into it — no drivers, no configuration, no cloud.
Inspiration
Send a robot to Mars, or to the bottom of the ocean — there is no cloud to call. A radio round trip to Mars is forty minutes; seawater eats radio for breakfast. Whatever intelligence the machine needs, it has to carry with it — and whatever hands it has, it has to discover on its own, because nobody is out there installing drivers.
Software agents already solved discovery: it's called MCP, the Model Context Protocol. But MCP stops at the network socket — it has never touched a wire. Every agent today only knows the tools someone hand-coded for it; plug in a sensor or a motor and the agent has no idea it exists. Hardware is invisible to agents. So I pushed MCP down to the firmware.
What it does
OpenCrawl is a fully offline voice agent whose capabilities are physical objects. Any device that speaks our one-page serial protocol announces its name, a logo, and a typed function table the moment it's plugged in — and each advertised function becomes a live LLM tool. Unplug it, and the agent doesn't just lose the ability; it loses the knowledge the ability ever existed.
- Fully offline pipeline on a Jetson Orin Nano: Whisper STT → Gemma (llama.cpp, tool-calling) → local TTS, with an animated robot face on a 5" panel. Speech in, hardware action, speech out — zero internet.
- Hot-plug discovery, live mid-demo: no drivers, no configuration, no restart. The device teaches the agent how to use it.
- Device 1 — Arduino Uno: GPIO control, an RGB status LED, and color-cycle animations. The rhythm runs on the microcontroller, not in the model — the agent delegates, the way your brain delegates your heartbeat.
- Device 2 — ESP32 gauge: a servo needle plus its own WiFi uplink. The agent stays offline; the device joins the network and trades on a live prediction market. One voice command chains three discovered tools — place a real bet. It also points the physical servo at the specified angle.
- An honest agent: tool results are ground truth. A 16-message sliding context with a live fullness meter, exact token accounting from the server's own tokenizer, and a hard rule that the agent says "that didn't work" instead of hallucinating success.
- Operator console: a desktop window into the agent's brain — the Jetson's telemetry, the live tool registry, scheduled routines and alerts, performance logs, even the system prompt.
How I built it
A Jetson Orin Nano runs three CUDA containers — llama.cpp serving Gemma 4 with all layers on GPU, Whisper STT, and local TTS — plus a Python agent that owns the mic, the face, the console, and a connection manager that watches serial ports for new devices. Firmware is C++ on two radically different chips — an ATmega328 with 2 KB of RAM and a WiFi-class ESP32 — both implementing the same manifest spec, with zero per-device code on the agent. The wire format is newline-delimited JSON-RPC 2.0 — the same format MCP speaks — over serial at 115200 baud, so the same frames work over USB today and Bluetooth tomorrow. The whole system survives a power pull: console autologin and supervised services bring the agent back with no keyboard attached.
Challenges I ran into
- No Ollama deployment existed for quantized Gemma 4, so I ditched Ollama and deployed from scratch: pulled the GGUF from the Hugging Face repo and served it with llama.cpp.
- A 2B model claims success on a few failed tool calls. I prompt engineered the tool responses to carry explicit
ok:falseerrors and wrote a truth-telling rule into the system prompt.
Accomplishments that Im proud of
The manifest protocol runs identically on completely different microcontroller architectures — an 8-bit chip with 2 KB of RAM and a WiFi-class ESP32 — with zero hand-written per-device code. The entire pipeline — discovery, inference, speech — runs with no internet connection at all. And the agent demos honestly: when hardware disappears or a tool fails, it says so.
What I learned
An agent that needs the cloud to act isn't autonomous — it's a peripheral of a data center. Proving genericity across architectures matters more than proving speed on any one of them. And small models can be trustworthy tool-users if you treat tool results as ground truth instead of suggestions.
What's next for OpenCrawl
More transports and more architectures: the same bytes over Bluetooth and bare serial lines, an ARM port, and devices that carry their own intelligence — a module running a local classifier should be indistinguishable, to the agent, from a servo. I'll publish the one-page spec so any maker can make their hardware agent-ready on a three-dollar chip. Every device will be an MCP server. I built the handshake.
Log in or sign up for Devpost to join the conversation.