• 

  Lipi

Lipi - Voice Dictation & AI Writing Assistant

Inspiration

I spend most of my day inside a terminal or an editor. I also talk to myself constantly - ideas, notes, commands, half-formed thoughts. I wanted a single hotkey that would turn any of that speech into clean text, already pasted wherever I needed it, without uploading audio to a cloud that I don't control.

Existing tools forced a compromise: local Whisper models are slow on CPU, cloud APIs are fast but opaque, and none of them were wired up to meeting transcription out of the box. When NVIDIA released Parakeet TDT 0.6B v3 - a model that runs at roughly $3\times$ real-time on a laptop GPU and can be shipped inside the app - and Nebius AI made Nemotron-3-Nano-30B-A3B available as a low-latency API endpoint for text tasks, the two pieces I had been waiting for were suddenly in place: local audio that never leaves the machine, and a cloud LLM fast enough to polish or summarise it in under a second.

What it does

Lipi (Bengali: লিপি, meaning "script") is a local-first desktop voice dictation app for Windows and Linux, built with Tauri and Rust.

• Press Alt + R from any app - your editor, browser, Slack, a terminal. Speak. Press it again.
• Lipi transcribes your voice locally using NVIDIA Parakeet TDT 0.6B v3 - audio never leaves your machine, and results come back in under a second even for longer clips.
• An optional AI polish step sends the raw transcript to NVIDIA Nemotron-3-Nano-30B-A3B on Nebius AI to fix grammar, adjust tone, convert speech to bullet points, or run a custom prompt - all configurable as plain Markdown presets.
• The finished text is auto-pasted into the window you were working in.
• Everything is saved to a local SQLite history you can search and replay.

Meeting Recorder

The second mode is a dedicated Meetings view. Lipi captures both microphone and system audio (loopback) simultaneously, feeds 30-second VAD-segmented chunks through Parakeet locally in real time, and displays a live rolling transcript. When you stop the meeting, Nemotron on Nebius reads the full transcript and returns a structured note with a short title, a plain-English summary of what was discussed, the key decisions the team reached, and a list of action items with due dates where dates were mentioned.

Every meeting session - including the full rolling transcript, the AI-generated summary, decisions, and action items - is automatically saved as a note in Lipi's local history. I can open any past meeting from the Notes view, search across all sessions, and re-send the transcript to a different AI preset or a coding agent later. Action items with due dates surface as desktop notifications so nothing falls through the cracks.

Our team has been running Lipi in daily standups and sprint reviews. The shared workflow is simple: one person starts the meeting recording on their machine, Parakeet transcribes in real time, and when the call ends Nemotron produces the structured note automatically. No manual copy-paste, no third-party meeting bot, no audio uploaded to a vendor.

How I built it

Audio pipeline (Rust / cpal)

Audio is captured at $16\text{ kHz}$ mono. For dictation, Silero VAD v5 (ONNX Runtime) detects the end of a phrase and triggers transcription. For meetings, the pipeline mixes microphone and loopback streams into a single PCM buffer, runs VAD, and emits chunks to the main thread via a channel.

Transcription - NVIDIA Parakeet TDT 0.6B v3 (local)

Parakeet runs entirely on the user's machine. Lipi bundles a unified local inference runner and manages model downloads automatically; the nvidia/parakeet-tdt-0.6b-v3 weights are fetched once and cached locally. Parakeet's token-and-duration (TDT) head delivers word-level timestamps at no extra cost. On a mid-range laptop GPU it processes a 10-second clip in well under a second - fast enough that dictation feels instant. Because transcription is local, audio never leaves the device.

AI Polish & Meeting Notes - NVIDIA Nemotron-3-Nano-30B-A3B on Nebius AI

After local transcription, if a preset is active, the raw text is sent to nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B via Nebius Token Factory's (tokenfactory.nebius.com) OpenAI-compatible chat completions endpoint. The model is a $30\text{B}$ MoE with only $3\text{B}$ active parameters ($10\%$ active compute ratio), so it is cheap and fast - typical polish of a 200-word transcript comes back in under a second. Preset prompts are plain Markdown files the user can edit.

For meeting summarisation, the entire session transcript is sent to Nemotron with a carefully crafted system prompt. The model returns a structured plain-text note covering the meeting title, a summary of what was discussed, key decisions reached, and action items with due dates where dates were mentioned. A small Rust function normalises the response and saves each part into the database separately.

Frontend (React + TypeScript + Tauri)

The UI is a single Tauri window. The main notes view keeps Alt + R history. The Meetings tab is a separate route that handles session state, live segment events over Tauri's event bus, and the review/action-items screen. A floating mini widget (Alt + M) stays always-on-top for quick access.

Storage

All notes, transcripts, meeting sessions, segments, and action items live in a local SQLite database via rusqlite. The only outbound network calls are the LLM requests to Nebius AI - transcription stays entirely on-device.

Challenges we ran into

1. Mixing mic + loopback into one transcription stream

System audio capture is platform-specific: WASAPI loopback on Windows, PulseAudio monitor sources on Linux. The two streams run at different sample rates and buffer sizes. Merging them into a single coherent $16\text{ kHz}$ PCM buffer without drift or glitching required resampling both streams independently with rubato and then mixing on a shared timeline.

2. VAD chunk boundaries vs. sentence boundaries

Silero VAD cuts on silence, which is correct for dictation but produces awkward mid-sentence chunks in meetings (people pause mid-thought). The current solution adds a small look-ahead: if a chunk ends with a conjunction or preposition, the segmenter waits for the next VAD boundary before closing the chunk. This is heuristic and still wrong sometimes, but dramatically reduces broken transcription lines.

3. Parsing Nemotron's meeting note response

Nemotron's meeting note output is free-form text - title, summary, decisions, and action items written in natural language. The Rust parser extracts each section by scanning for known headings and bullet patterns, then stores them as separate database fields. If parsing produces nothing useful, the raw text is saved as the summary so the note is never lost.

4. Cross-platform auto-paste

Pasting into the previously focused window requires sending Ctrl + V via platform APIs - xdotool / ydotool on Linux (with Wayland needing an extra desktop portal call), and the Win32 SendInput API on Windows. Keeping this reliable across display servers (X11 vs. Wayland), session types, and app sandboxing took more iteration than any other part of the project.

What I learned

• Parakeet TDT running locally is genuinely fast enough to keep up with a live meeting. On a mid-range GPU, 30-second chunks are transcribed in 1–2 seconds - well within the VAD pause window, and with zero network latency.
• Nemotron on Nebius is a cost-effective summariser. The $3\text{B}$-active-parameter MoE profile means meeting notes for a 1-hour call cost a fraction of a cent.
• Audio is the hardest part of any voice app. The transcription model is almost the easy bit; getting clean, merged, properly-timed PCM off two different OS audio subsystems is where the real engineering lives.
• OpenAI-compatible APIs make provider switching trivial. Lipi can point at cloud provider, or a local server with a single URL change - no SDK coupling.

Accomplishments we're proud of - Real-world usage

Lipi isn't a prototype - the whole team uses it every day. We run it in standups, sprint reviews, and async voice notes. The workflow is zero-friction: one hotkey to record, one hotkey to stop, and the polished text or structured meeting note is already in the right place. Parakeet's local speed means there's no waiting; Nemotron on Nebius handles the heavy thinking - grammar, tone, and meeting summaries - without any of us having to think about prompts. Replacing manual meeting minutes has saved each team member 15–20 minutes after every call.

What's next for Lipi

• Word-level timestamps from Parakeet rendered in the meeting transcript (the TDT head already returns them; the UI just doesn't display them yet).
• Speaker diarization - a second pass over the meeting audio to label turns, which Nemotron can then format as a proper dialogue transcript.
• Agent dispatch mode - send the edited dictation transcript directly to Claude Code, Cursor, or Agy as a headless prompt, turning Lipi into a voice cockpit for coding agents.

---

The codebase is fully open-source and available at github.com/debjit/lipi, with builds and documentation at lipi.debjit.in.

Built With

• api
• react
• sqlite
• tauri
• typescript