CommunityOS

A conversational agent living in your community — knows your voice, remembers every visit, greets you by name. Household-scale org, persistent memory, a face that looks at you. Nemotron + Parakeet.

Comment

Inspiration

Coexistir is a real intentional community. It runs on the things most communities run on: meals, favors, a shared kitchen, and memory. Who cooked. Who kept their word. Who was at the table when the thing was decided. None of that lives in a database. It lives in people, and in scrollback.

Two failures kept showing up. Translation apps assume a personal device in the conversational path: the phone becomes the addressee, turn gaps stretch from 200 ms to ten seconds, and whoever holds it holds the floor. And AI assistants reset every session, so a community's memory ends up in a chat log and in whoever was paying attention.

CommunityOS is the opposite bet. A screen in the middle of the table, belonging to nobody. A voice in the map that recognizes you when you speak. A record that is a folder of Markdown and a git history — readable without our software, exportable without our permission. The system carries words and keeps only what a person deliberately hands it.

The project predates the hackathon. During the submission period (Aug 26 – Oct 30, 2026) it grew from a translation aid into a system: a voice-first front door with a Live2D avatar, a bilingual PT/EN voice loop with barge-in, speak-to-enroll voice identity (voiceprint-bound names, strangers stay unknown), per-person memory threads, one Owl publishing voice and face, a Markdown-and-git record, and an NVIDIA Nemotron polish lane on the manager's replies.

What it does

Three surfaces and one trust layer: someone speaks, the system carries the words, and only what a person files becomes memory.

Mesa — the table that translates

One screen on the table captures; every other device is a reader. Speech is transcribed while people code-switch, translated, and shown with the original beside it. The hosted Deepgram path hears ten languages; the local Parakeet path hears twenty-five; twenty-seven can be read. PAUSE clears every board. The board holds five lines with a TTL, audio is deleted after transcription, and a failed translation never costs the table the line — the original stays, the second column visibly does not arrive.

A day at the table can settle into a curated artifact, with the verbatim one tap away. The bot is shipped; the table has not yet been used at a real lunch; the artifact composer is built and covered by 25 tests.

Owl — the voice in the room

Owl is a LiveKit agent in the Coexistir room. Talk to it and it answers out loud, publishes a speaker label, and files the turn into MongoDB and a per-person Graphiti thread. Its face is a Live2D Hiyori rendered in headless Chrome and published as a video track from the same participant — the single-Owl build, not a voice and a puppet.

Today the agent does not read those threads back, and it does not greet you by name. The room recognizes the voice; the conversation stays generic. That is the next seam, not a claim.

Voice identity — claims, not names

Enrollment produces a 192-dimensional voiceprint with an ERes2Net model running on CPU through ONNX Runtime. Identification is cosine similarity with a hard margin rule: if the top two scores are closer than 0.15, the label is stranger even when the top match looks good. A guest print is never trusted. A name is a claim, not a rename: one steward confirmation or two distinct people claiming the same name verifies it; a single self-claim stays unverified and is displayed that way.

The meeting view plots live voices on a 2D map from their embeddings and lets you play a sample; a 607-chunk, three-speaker corpus was used to validate the geometry.

The record — memory the community owns

/relato and /tarefa file a Markdown document with front matter into a git repository, one commit per filing. The record is readable with no software: open the folder, read the files, read the history. A steward page shows what is open in the reader's language over the original, with one button to close. The brain answers only from record tools and prints the ids of what it read; it cannot write. A tap files, or nothing does.

Multi-tenant isolation is specified, not built: one directory and one git repository per community, with traversal validated at the funnel.

The two memories are not yet one memory. Owl writes turns to MongoDB and Graphiti; the record's brain reads the git record. They share a room, not a recall path.

How we built it

  • World. Castalia (MIT, private today): Phaser 3 + React + Colyseus + LiveKit. A 2D gameworld with a guest entry, not a meeting grid. Walk into the Coexistir map; the room is a place.
  • Voice loop. Node @livekit/agents 1.9.3, Deepgram Nova-3 ears, nvidia/nemotron-3-nano-30b-a3b via NVIDIA NIM on the composed-turn polish lane, Deepgram Aura-2 Luna TTS. No fine-tune; prompt engineering only — deterministic compose from the household task rows, Nemotron rephrase-only (grounded rephrase 558 ms TTFT / 626 ms total; ungrounded answers hallucinate, so it never answers from scratch).
  • Voice identity. A Python stdlib HTTP sidecar on 127.0.0.1:3124: ONNX Runtime + numpy + soundfile, ERes2Net 192-d, 16 kHz, 80-mel log frontend, cosine. No torch, no network, no API key. The live health endpoint reports eres2net-voxceleb-192d and threshold 0.45.
  • Memory. MongoDB castalia.utterances for the log; Graphiti /add_memory for per-person threads (coexistir-owl-<who>). Writes are fire-and-forget, so a dead graph never drops a call.
  • Mesa. Python with a stdlib-only hosted path; Deepgram / Parakeet / Whisper for speech; Google / Azure / LLM / DeepL / local for translation; a Redis board with a TTL; vanilla JavaScript. The Village OS core's 234 tests cover this path along with the record.
  • Face. Headless Chrome renders the /owl Live2D page; CDP screencast frames go into an rtc-node VideoSource; one hiyori track at ≤15 fps and 500 kbps. After throttling, the publisher is about 6% of a 24-core box.
  • Deploy. Self-hosted LiveKit, Castalia behind Cloudflare, Mesa on Vercel. The whole loop is one room and one table.

Challenges we ran into

  • Crosstalk, and a sample-rate bug underneath it. One blended transcript is worse than none. The first 48→16 kHz path took every third sample with no low-pass, folding 8–24 kHz energy into the voice band as score jitter. The fix is a 15-tap windowed-sinc FIR at 7.2 kHz, then decimate by three. It is the least glamorous line in the repo and it changed the labels.
  • A confident wrong name. A 0.61 cosine match to the enrolled guest print, with a margin of 0.019, is exactly the kind of number that talks you into naming someone. The margin rule forced stranger anyway. We made the margin strict — under 0.15 is a stranger — and then made names claims: two distinct claimants or a steward, never a self-assertion. The hard part was accepting that “unknown” is a better product than a plausible guess.
  • Code-switching. Language ID on audio is the first thing to break on accented speech. We detect on the text, keep the speech model's guess as a tiebreaker, and accept a closed choice between the languages at the table. Ten with code-switching beats thirty-six without. Vocabulary repair is deliberately conservative: hora was becoming horta; marinna resolved to Marina while Mariana sat just below the threshold. Ambiguity now leaves the word alone.
  • Privacy drift. A third column, updated_at, survived a migration and a test that only counted columns. It was written on every change and read by nothing. Removed. A VILLAGE_OS_LOG_CONTENT flag printed a warning at boot and gated nothing; removed. The test now names the columns it expects, because counting them is how the column got through. A record of who the machine understands least is a record of whose Portuguese is worst.
  • The demo is not the system. A guest entry that loops, a tunnel that rotates URLs, a headless WebGL readback that returns black, an adaptiveStream subscriber starved when hidden. Each looked like a product bug and was an environment bug. We encoded them as checks, not anecdotes.

Accomplishments that we're proud of

  • 234 tests green in the Village OS core (5.51 s), including 25 for the artifact composer, plus 15 on the name-claim policy in Castalia. The privacy rules are tests, not intentions.
  • A closed loop on a real call: speech → speaker label → MongoDB row + coexistir-owl-* Graphiti thread, with the same transcript and score on both sides.
  • One Owl, voice and face: the single-participant build publishes both from one participant identity; a cutover run fed 2.14 million video frames at ≤15 fps and about 6% CPU on a 24-core box. The transport still drops, so this is a build to finish, not an uptime claim.
  • A voice map with real geometry: 607 chunks, three clusters, silhouette 0.53, centroid distances 0.73–0.86. Separately, in the enrollment sweep, 1,195 candidate clips were screened, 141 silent clips skipped, and two voiceprints enrolled.
  • A table designed for a noisy kitchen: ten code-switched languages on the hosted path, a five-line board with a TTL, PAUSE that clears every screen, and a failed translation that never deletes the original line.
  • A record that outlives the app: Markdown + git, one commit per filing, readable with ls and cat. Export is a folder, not a support ticket.

What we learned

  • Form factor is the product. A tablet in the middle of the table, belonging to nobody, is a different object from a phone held by the person who needs help. The same software becomes the thing nobody uses when it is aimed at people.
  • Errors must be visible and cheap. The original travels with every translation. Warnings appear sometimes, on purpose, because a disclaimer on everything is wallpaper. No model ever rewrites a transcript; the bot carries words, it does not produce them.
  • Identity is the whole game. In community software, “who said that, really?” consumed more design than any model choice. The correct answer is often “unknown”, and the product has to make unknown look deliberate.
  • Privacy is a schema property, not a policy. Name the columns. Delete the unused flag. Never log content. A store that grows is a recording, whatever the code around it says.
  • Models are last. Microphone quality beats model choice; the village's own vocabulary beats a bigger model; context beats a longer context window. The model is the last place to spend.
  • A demo can be live and still not be a system. The system is the record, the exit, and the ability to say no. That is the part we are still building.

What's next for CommunityOS

  • Measure the wired Nemotron lane on a real conversation. Owl's composed-turn polish lane runs on nvidia/nemotron-3-nano-30b-a3b via NVIDIA NIM (grounded rephrase 558 ms TTFT / 626 ms total); next is latency over a full real conversation.
  • Make the repo and the demo public. MIT license, setup instructions, a judge path, and the three-minute video below. Repo: https://github.com/liminalcommons/coexistir-door.
  • Use the table at a real lunch. Compare the curated artifact with the verbatim, and tune the warning threshold from what people actually experience rather than from a benchmark.
  • Make the two memories one memory. Read the Graphiti thread back into Owl's prompt, with the same citation discipline as the record brain: the ids travel with the sentence, and the agent still cannot write.
  • Build the tenancy that is only on paper. One directory and one git repository per community, then onboard a second community that is not Coexistir. The pattern should port anywhere people share meals and promises.
  • Finish the voice map. Let it grow as the room talks, instead of seeded snapshots, and finish the one-Owl cutover so there is no twin publisher left to explain.

Built With

Share this project:

Updates

Submission history