localscribe

LocalScribe is a privacy-first, on-device AI application that transforms voice recordings into accurate transcripts, concise summaries, and actionable notes

Comment

Inspiration

Many AI transcription and summarization tools require uploading sensitive recordings to cloud services, raising privacy concerns and making them unusable without an internet connection. We wanted to prove that modern AI can be both powerful and private by building an application that performs speech recognition and summarization entirely on-device. LocalScribe demonstrates that users can keep complete ownership of their data while still benefiting from fast, intelligent AI.

What it does

LocalScribe is a privacy-focused offline voice assistant that converts audio recordings into transcripts, summaries, and action items without making a single network request.

Users can record or upload a voice note, meeting, lecture, or interview, and LocalScribe will:

  • Transcribe speech into text using Whisper.cpp.
  • Generate concise summaries using a quantized large language model running through Llama.cpp.
  • Extract key discussion points and actionable tasks.
  • Run completely offline, even in Airplane Mode.
  • Keep all recordings and generated notes on the user's device.

Because everything runs locally, sensitive information never leaves the device.

How we built it

LocalScribe combines two high-performance, open-source AI inference engines optimized for Arm processors.

  • Whisper.cpp performs offline speech-to-text transcription.
  • Llama.cpp runs a quantized GGUF language model to summarize transcripts.
  • Python orchestrates the end-to-end processing pipeline.
  • Gradio provides a lightweight user interface for uploading or recording audio.
  • Quantized language models (Q4_K_M and Q8_0) were evaluated to balance speed, memory usage, and output quality.

The processing pipeline is straightforward:

  1. Capture or upload audio.
  2. Transcribe audio locally.
  3. Generate summaries and action items locally.
  4. Display results with processing times.

No cloud APIs, external servers, or internet connectivity are required.

Challenges we ran into

One of the biggest challenges was getting multiple AI models to work together efficiently on resource-constrained hardware. We also had to carefully choose model sizes that balanced performance, memory consumption, and summary quality.

Another challenge was ensuring compatibility across different Arm devices while maintaining a smooth user experience. Model quantization required experimentation to determine which versions offered the best speed improvements without noticeably reducing output quality.

Accomplishments that we're proud of

We're proud that LocalScribe demonstrates practical, real-world AI that genuinely protects user privacy.

Highlights include:

  • Successfully running speech recognition and summarization entirely offline.
  • Achieving fast inference using quantized language models.
  • Creating an end-to-end workflow with zero network calls.
  • Building a solution that works even in Airplane Mode.
  • Showing how efficient on-device AI can replace cloud-based processing for everyday tasks.

What we learned

Building LocalScribe gave us a much deeper understanding of on-device AI inference, model quantization, and performance optimization on Arm hardware.

We learned how significantly quantization can reduce memory usage while maintaining strong output quality, and we gained practical experience integrating multiple AI models into a single offline workflow.

Most importantly, we learned that privacy and AI do not have to be competing priorities—careful engineering allows both to coexist.

What's next for LocalScribe

Our roadmap includes several exciting improvements:

  • Real-time live transcription during conversations and meetings.
  • Speaker diarization to distinguish multiple speakers.
  • Automatic translation into multiple languages.
  • Searchable local knowledge base of previous transcripts.
  • Calendar and task manager integrations for extracted action items.
  • Android and iOS native applications.
  • GPU and NPU acceleration for even faster inference on modern Arm devices.
  • Enterprise deployment for organizations that require complete data privacy.

Our long-term vision is to make LocalScribe the go-to offline AI productivity assistant for students, professionals, journalists, healthcare workers, and anyone who values privacy without sacrificing the power of artificial intelligence.

Built With

Share this project:

Updates