WhisprDoc

Multilingual Voice-to-Text AI for Smart PDF Filling Just talk. WhisprDoc does the rest.

Comment

🎙️ WhisprDoc

Just Talk. WhisprDoc Handles the Rest.

AI-powered multilingual voice-to-PDF automation for everyday users. Speak naturally in over 30 languages, and WhisprDoc fills complex PDF forms automatically while maintaining version history, autosaving every change, and preserving user privacy with on-device speech recognition.

🌍 Inspiration

Paperwork shouldn't be a language barrier for anyone.

Every day, people fill out healthcare forms, tax documents, immigration paperwork, school applications, and legal contracts—often in languages they may not fully understand. The process is slow, repetitive, and especially frustrating for non-native speakers.

I wanted to remove that friction for individuals and knowledge workers, like myself, who spend hours completing document-heavy workflows.

Instead of making users understand PDFs, we wanted PDFs to understand users.

WhisprDoc combines on-device speech recognition, multilingual translation, AI semantic reasoning, and intelligent PDF field mapping into a single workflow that lets anyone simply talk while the application handles everything else.

✨ What it does

WhisprDoc lets users:

  • 🎤 Speak naturally instead of typing

  • 🌎 Automatically translate over 30 languages into English

  • 📄 Fill virtually any PDF without templates

  • 💾 Autosave every pause while speaking

  • ⏪ Restore previous document versions instantly

  • 🔒 Keep raw speech processing on-device for privacy

🏗️ System Architecture

Our architecture is intentionally split into two completely separate AI pipelines.

The first pipeline understands the PDF.

The second understands the user's speech.

Separating these responsibilities dramatically improved reliability, debugging, and scalability.

Architecture Overview


React Frontend

      │

      ▼

On-device WhisperAI

      │

      ▼

Next.js Server

      │

      ├────────────► Supabase Auth

      ├────────────► Amazon S3 / Supabase Storage

      │

      ├────────────► Pipeline #1

      │

      ├────────────► Pipeline #2

      │

      ▼

OpenRouter AI

      │

      ▼

Structured JSON

      │

      ▼

AWS DynamoDB

📑 Pipeline 1 — Document Understanding

The first pipeline only runs when a user uploads a new PDF.

Its goal is to understand the document itself.

Instead of hardcoding templates for thousands of forms, we extract every widget, label, and visible piece of text before converting each page into HTML.

The HTML representation preserves spatial relationships much better than raw PDF text, allowing the language model to understand which labels belong to which form fields.

OpenRouter then converts this information into structured JSON containing semantic field names that become reusable for the rest of the document session.

Pipeline 1

Pipeline Steps

  1. Upload PDF

  2. Extract widgets & text using pdf.js and pdf-lib

  3. Convert pages into HTML

  4. Send HTML to OpenRouter

  5. Receive semantic JSON field map

After this pipeline finishes, every field in the document has an AI-generated meaning rather than just an internal PDF identifier.

🎤 Pipeline 2 — Voice → Filled PDF

Once the document has been mapped, the second pipeline begins.

This pipeline converts natural speech into completed PDF fields for everyday users and knowledge workers.

Unlike traditional speech-to-text systems, speech recognition never leaves the user's computer.

WhisperAI runs locally inside a WebWorker using Hugging Face Transformers.

Only the transcript—not the audio—is sent to OpenRouter.

The language model determines user intent, translates non-English speech into English, fixes common transcription mistakes, and produces structured JSON matching the semantic field map generated earlier.

That JSON is then merged directly into the PDF.

Pipeline 2

Pipeline Steps

  1. User speaks

  2. WhisperAI performs local speech recognition

  3. Transcript generated

  4. Transcript sent to OpenRouter

  5. Structured JSON returned

  6. JSON mapped onto PDF fields

  7. Finished PDF generated

☁️ Backend Architecture

Our backend is built using Next.js Server Actions.

The server is responsible for:

  • Authentication

  • Secure API calls

  • PDF parsing

  • OpenRouter requests

  • JSON validation

  • PDF generation

  • Version history

  • Autosave synchronization

Authentication is handled by Supabase Auth while uploaded PDFs are stored in Amazon S3 / Supabase Storage.

⚡ Why DynamoDB?

Choosing DynamoDB was one of the biggest architectural decisions of the project.

Most PDF applications save only once after the user finishes editing.

WhisprDoc autosaves after every voice pause and every keystroke for users, including professionals and knowledge workers handling document-intensive tasks.

That means our application performs roughly 10× more writes than a traditional PDF editor.

A relational database with row-level locking would quickly become a bottleneck.

Instead, DynamoDB's partition design allows every document write to remain independent.

Using:

  • Partition Key: User ID

  • Sort Key: Document ID / Timestamp

provides:

  • Single-digit millisecond writes

  • Massive horizontal scaling

  • No row-lock contention

  • Natural document history

Even better, version history wasn't something we had to bolt on later.

Because every save creates a new immutable record, users can literally rewind their document to any previous point in time.

DynamoDB's flexible NoSQL structure was also perfect because every PDF produces a completely different JSON schema.

🔄 Autosave Architecture

Every pause while speaking immediately creates:

  • Updated document fields

  • Autosave snapshot

  • Immutable revision

  • Audit log entry

This provides Google Docs–style reliability while editing PDFs.

Even browser crashes or accidental refreshes cannot lose work.

🌐 Multilingual Support

WhisprDoc currently supports over 30 spoken languages.

If the user speaks:

  • Spanish

  • Hindi

  • French

  • Mandarin

  • Arabic

  • Japanese

  • (and many more)

WhisperAI first transcribes the speech.

OpenRouter then automatically detects the language and converts the transcript into clean English before extracting structured data.

This allows English PDF forms to be completed by users speaking entirely different languages.

🛠️ Built With

  • Next.js

  • React

  • TypeScript

  • Tailwind CSS

  • Hugging Face Transformers

  • WhisperAI

  • Web Workers

  • OpenRouter

  • pdf.js

  • pdf-lib

  • AWS DynamoDB

  • Amazon S3

  • Supabase

  • Supabase Auth

  • Vercel

  • Lucide React

🚧 Challenges We Faced

Semantic PDF Mapping

PDFs are notoriously inconsistent.

Many forms contain fields like:


Text1

Field_4

Textbox12

These names have no meaning.

Converting entire documents into HTML before sending them to the language model dramatically improved semantic understanding.

Database Design

Traditional SQL databases perform well for occasional writes.

WhisprDoc performs continuous writes.

Designing around DynamoDB allowed autosave, revisions, and version history to become core features rather than expensive add-ons.

Database Schema

Supporting Every PDF

Instead of creating templates for every government form, healthcare form, school application, or tax document, we designed a generic semantic mapping pipeline capable of understanding arbitrary PDFs for everyday users.

📚 What We Learned

Building WhisprDoc taught us that AI is most powerful when broken into specialized stages rather than asking one giant prompt to solve everything.

Separating document understanding from speech understanding made every component easier to improve independently.

We also gained a much deeper appreciation for distributed NoSQL database design. Autosaving continuously changes the architecture entirely, and DynamoDB proved to be an excellent fit for high-frequency immutable writes.

Finally, we learned that privacy matters for everyday users. Running Whisper locally means users never upload raw audio, significantly reducing latency while increasing trust.

🚀 Future Plans

  • Agentic Form filling

  • More languages

  • Offline document filling

  • OCR for scanned PDFs

  • Consumer healthcare workflows

  • Legal document automation for individuals

  • Family / shared document collaboration

  • AI-powered document validation

  • Mobile application

❤️ Closing Thoughts

Paperwork and language barriers remain two of the largest bottlenecks in everyday life—especially for individuals navigating healthcare, legal, finance, education, and immigration systems.

WhisprDoc doesn't just translate speech.

It understands documents.

By combining local speech recognition, semantic document analysis, AI reasoning, and scalable cloud infrastructure, we created an experience where users can simply talk—and the paperwork completes itself.

Closing

Built With

  • aws-dynamodb
  • hugging-face
  • next.js
  • openrouter
  • react
  • shadcn/ui
  • supabase
  • tailwind-css
  • typescript
  • v0-(vercel)
  • vercel
  • whisper-ai
  • zod
Share this project:

Updates