Inspiration

People make small commitments constantly: “I’ll send you the file,” “I’ll call them tomorrow,” “I’ll look into it.” Most of these are never written down because stopping a conversation to create a note or task is disruptive. Then the commitment gets forgotten.

We already have voice recorders, transcription tools, meeting summaries, and task managers, but they mostly solve only part of the problem. Recording everything still leaves the user with hours of audio to process, and identifying a commitment is not the same as actually completing it.

We wanted to build something around a simple idea:

What if an AI could listen for what you said you would do, remember it, and help turn it into something that actually gets done?

That idea became LYZN.

What it does

LYZN is a voice-first AI system built around a small Bluetooth wearable, with the phone microphone available as an alternative capture method.

A recording moves through the system from capture to execution:

Capture → Transcribe → Understand → Remember → Approve → Act → Verify

The wearable records speech locally. The mobile app synchronizes recordings, removes unnecessary silence before upload, and sends the audio through a cloud processing pipeline.

LYZN uses two speech-to-text systems — Deepgram and Sarvam — and reconciles their outputs before sending the transcript to Claude through AWS Bedrock.

Claude analyzes the conversation to generate a summary, identify topics, extract commitments and action items, and store useful information in long-term memory.

Extracted tasks remain under human approval. Once approved, they can be sent to a desktop daemon running on the user's computer, which performs the task and returns a receipt containing the execution result.

LYZN therefore goes beyond “remembering what was said.” It connects conversation to action and provides a verifiable record of what happened.

How we built it

LYZN is made up of five connected subsystems:

Wearable: a Bluetooth voice recorder used for audio capture.
Mobile app: React Native + Expo for pairing, recording synchronization, uploads, transcripts, tasks, receipts, and AI chat.
Website: React + Vite + TypeScript, including an interactive 3D pendant experience.
Backend: Go + Fiber running on AWS Lambda, with DynamoDB, S3, SQS, and AWS Bedrock.
Desktop app: Electron, running a background daemon that polls for approved tasks and executes them locally.

The project contains roughly 560 files and ~45 backend endpoints. The three client applications share a common design package so that the web, mobile, and desktop interfaces use the same visual system.

The AI pipeline combines several components:

Deepgram for transcription, timestamps, and speaker diarization.
Sarvam for a second transcription pass, particularly useful for Indian languages and accents.
Claude for transcript enrichment, commitment extraction, and long-term-memory selection.
GitLoom for persistent searchable AI memory.
Server-Sent Events for streaming AI chat.
On-device voice-activity filtering to remove non-speech before upload.
Human-in-the-loop approval before autonomous task execution.

Challenges we ran into

One of the hardest problems was the wearable itself. The manufacturer-provided Bluetooth protocol documentation did not accurately match the behavior of the firmware on the hardware we had. We had to probe the device directly, reverse-engineer the working protocol, and make LYZN detect and adapt to different command dialects.

We also had to design for failure rather than assuming everything would work perfectly.

For desktop task execution, a machine can crash, sleep, disconnect, or lose its network connection in the middle of a task. LYZN uses a 15-minute lease-based claiming system so abandoned tasks can automatically return to the queue.

Task completion is also idempotent and atomic: retrying a completion request does not create duplicate receipts.

Other difficult areas included synchronizing audio across the wearable and phone, reconciling two independent transcription systems, streaming AI responses safely while they were still being generated, and keeping authentication and payment operations server-authoritative.

Accomplishments that we're proud of

We are especially proud that LYZN became a complete working system rather than a thin AI wrapper.

The most important accomplishment is the full path from a real-world conversation to an executable action:

speech → recording → transcription → AI understanding → commitment extraction → human approval → computer execution → receipt

We also built several pieces that required solving problems outside the usual AI API layer:

Reverse-engineering a real Bluetooth device whose documentation did not match its firmware.
Reconciled transcription using two independent speech-to-text engines.
Built persistent conversational memory instead of relying only on raw transcripts.
Designed lease-based recovery for distributed task execution.
Made task completion idempotent under network retries.
Built a shared design system across React, React Native, and Electron.
Added a graceful static fallback for the website's 3D experience so the page remains usable on slower devices.
Implemented streaming AI chat with safe incremental markdown rendering.

What we learned

The biggest lesson was that building an AI product is not just about prompting a model.

The model is only one part of the system.

We had to think about:

input quality → structured information → memory → decisions → execution → failure recovery → verification

We also learned how important failure handling is when AI is connected to real actions. A system that works only on the happy path is not enough once it can affect someone's files, applications, or workflow.

Working with real hardware taught us another lesson: documentation is not always reality. We had to test assumptions against the physical device and build around what the hardware actually did.

Most importantly, we learned that AI becomes much more powerful when it is connected to a reliable system around it. Instead of simply summarizing conversations, LYZN explores how AI can understand what was said, retain the useful context, propose actions, and safely connect those actions to execution.

What's next for Lyzn

The next stage is to make the system capable of handling a wider range of real-world workflows while keeping the same human-approval model.

Planned directions include:

Expanding the number of task categories the desktop agent can safely execute.
Supporting more wearable hardware variants as their protocols are understood.
Building deeper memory and context capabilities on top of the existing persistent memory system.
Continuing to improve the desktop execution layer and its background engine.
Exploring more ways for LYZN to connect conversational context with useful actions without removing the human from the approval loop.