Zipity

Dashboard
Assistant

Inspiration

Students are expected to manage classes, assignments, goals, notes, and AI chats across disconnected tools. Most assistants also forget who the learner is between sessions. Zipity is built around a simpler idea: one personal learning companion that grows with a student from the beginning of school through graduation.

What it does

Zipity is a local-first desktop learning companion that:

creates a personalized learner profile during first-run onboarding
organizes multiple subjects, assignments, and learning goals in one place
remembers relevant learner context while keeping it scoped to the device owner
streams clear, structured answers with headings, lists, tables, quotes, and code blocks
supports fast voice conversations with ElevenLabs
wakes hands-free when the student double-claps, then listens for a request
uses the name Zipity throughout the product for a consistent companion experience

The current demo profile is prefilled with realistic Grade 10 subjects and tasks so judges can explore the full workflow immediately.

How we built it

The desktop client uses Tauri 2, Rust, React 19, TypeScript, and Vite. A Rust/CPAL audio pipeline performs continuous low-overhead microphone analysis with an adaptive noise floor and double-clap detection. The backend uses FastAPI, PostgreSQL, and Redis, with typed contracts shared across the workspace.

For the live assistant, Zipity uses Baseten to serve the GPT-OSS 120B model and ElevenLabs for speech. Responses stream into a safe structured renderer, while speech output receives a clean text-only version for natural delivery.

How Codex and GPT-5.6 accelerated the build

Codex with GPT-5.6 acted as the engineering partner across the build: it mapped the monorepo, traced the desktop-to-API request path, implemented the personalized onboarding and owner-scoped memory flow, hardened the Rust wake detector, redesigned the companion interface, generated the Zipity icon pipeline, and repeatedly ran the application to diagnose real microphone and desktop-runtime failures.

The most important architectural decisions were made through those Codex sessions:

keep identity and learning context explicitly owner-scoped instead of relying on one global chat history
run clap detection natively in Rust for lower latency and more reliable Windows audio access
separate display formatting from speech formatting so rich answers remain readable without making TTS sound robotic
use frame-batched streaming updates to keep long answers smooth
validate changes with API, UI, Rust, and production-build checks instead of stopping at generated code

Codex also shortened the debug loop by inspecting logs, reproducing failures, editing across Rust/Python/TypeScript boundaries, and verifying each fix in the running desktop app.

Challenges we ran into

Reliable hands-free wake-up was the hardest problem. Real microphones have changing ambient noise, device contention, and very different clap signatures. We built adaptive thresholds, timing windows, diagnostics, and recovery paths rather than treating a clap as a fixed-volume spike.

We also had to balance rich, structured assistant responses with low-latency streaming and natural speech. That led to a two-path renderer: safe Markdown-like presentation for the screen and stripped, speakable text for voice.

Accomplishments that we're proud of

a working native desktop companion rather than a chat mockup
personalized, owner-scoped context across many school subjects
adaptive double-clap wake detection
responsive streaming with structured, safely escaped answers
an end-to-end voice loop using ElevenLabs
broad automated coverage across the FastAPI service, React interface, and Rust desktop layer
a cohesive, minimal visual identity built around Zipity's brain-and-lightning mark

What we learned

A believable AI companion is not only a model call. It needs identity boundaries, reliable input, useful long-term context, graceful failure states, and an interface that makes complex answers easy to scan. Native audio and streaming UI work also benefit enormously from tight observe-edit-test loops—the kind of workflow Codex enabled throughout this project.

What's next for Zipity

Next we plan to add opt-in encrypted sync, a longitudinal learning graph, curriculum and calendar integrations, richer teacher/guardian controls, packaged installers, and a hosted backend so students can carry their context safely across devices.