TranscribeNya

Find suspicious transcript segments and review safe local corrections before anything changes.

Comment

Inspiration

After transcribing long audio or video files, reviewing everything from beginning to end to find possible errors was very inefficient for me.

At the same time, in my early experiments with fully automated correction, the system sometimes changed text that should not have been changed.

I therefore designed the tool to help users find potentially problematic sections while leaving the final decision to accept or reject a correction to the user.

What I added during Build Week

TranscribeNya existed before Build Week with local Whisper transcription, subtitle editing, and warning detection.

During Build Week, I added job-specific vocabulary, Pocket Expert, local RAG, a candidate-selection LoRA adapter, a human-review UI, and the associated evaluation and packaging workflow.

What it does

TranscribeNya transcribes audio and video files locally using Whisper.

For this project, I added the ability for users to register custom vocabulary, including readings and preferred spellings.

Potentially incorrect sections and sections related to registered vocabulary can then be sent to Pocket Expert for local review.

Pocket Expert uses local RAG and a candidate-selection model to choose one of the following actions:

  • Keep the original text
  • Replace it with a specific candidate
  • Mark it as uncertain

Before applying a suggestion, the user can review the candidate and the reason why the section was flagged in the UI. Rejecting a candidate does not change the transcript.

Even when a candidate is accepted, only the text range explicitly selected by the user is replaced.

Pocket Expert’s main review process runs offline and does not call the GPT-5.6 API at runtime.

How we built it

  • Electron desktop application
  • TypeScript and React
  • Local media processing with Whisper and FFmpeg
  • Local RAG for custom vocabulary and review context
  • MLX runtime
  • Qwen3-1.7B-MLX-4bit base model
  • Candidate-selection LoRA adapter
  • Structured candidate decisions instead of free-form generation

How I used Codex and GPT-5.6

I used Codex throughout the project for architecture planning, Electron and UI implementation, IPC and local inference integration, regression tests, packaging, and end-to-end verification.

I used GPT-5.6 through Codex to prepare and expand teacher-data candidates, build evaluation cases, analyze model outputs, and compare candidate-selection experiments.

GPT-5.6 was used during development. TranscribeNya does not call the GPT-5.6 API at runtime.

Challenges we ran into

In the initial free-form LoRA experiment using 30 source cases, the model produced incomplete corrections and corrections that covered too much text. This showed that the system needed to avoid making changes when it was uncertain.

Even if the overall score improved, I could not adopt a model if its performance on “keep” and “uncertain” cases became worse.

The 100-source-case experiment improved replacement accuracy, but its independent evaluation did not meet the safety requirements for “keep” and “uncertain” cases.

Packaging a local desktop application also required bundling a 1.7B-class LLM, native binaries, and platform-specific verification.

Evaluation / Results

  • Adopted fixed 11-case evaluation
    • Baseline: 2/11
    • Candidate-selection LoRA: 10/11
    • Adopted for the MVP
  • Holdout evaluation of the LoRA trained with 100 source cases
    • Evaluated on 20 records created by expanding 10 source cases that were not used for training into two conditions each
    • Baseline: 6/20
    • LoRA: 16/20
    • Replace cases: 12/12
    • Keep cases: 4/6
    • Uncertain cases: 0/2
    • Not adopted as the default model because it did not meet the safety requirements
  • Additional verification
    • End-to-end testing of the packaged Electron application
    • Offline inference testing
    • UI testing with real audio
    • Verification that rejecting a candidate leaves the transcript unchanged
    • Verification that only the selected text range is replaced

Accomplishments that we're proud of

  • Made Pocket Expert’s main review process run entirely locally.
  • Implemented a candidate-based correction UI.
  • Added explicit accept and reject actions.
  • Prevented automatic replacement across the entire subtitle.
  • Designed the system to preserve the original text when a candidate is rejected.
  • Verified the workflow in the packaged Electron application.
  • Documented both the adopted model and the larger experiment that was not adopted.
  • Prepared repository-only verification and separate verification instructions for tests that require a physical macOS environment.

What we learned

At first, I focused on improving accuracy, but the overall accuracy score alone was not enough.

Even if replacement accuracy improves, the system cannot be used in practice if it also changes text that should not be changed.

I learned that “replace,” “keep,” and “uncertain” need to be evaluated separately.

What's next for TranscribeNya

  • Improve the safety gates for “keep” and “uncertain.”
  • Expand the evaluation cases while maintaining an independent holdout dataset.
  • Use review history from real-world use to improve candidate ranking.
  • Expand the warning explanations and validate them with more real-world transcription cases.
  • Improve distribution, signing, notarization, and setup so the application can be used on a wider range of Macs.

Built with

  • Electron
  • TypeScript
  • React
  • Whisper
  • FFmpeg
  • MLX
  • Qwen3-1.7B-MLX-4bit
  • LoRA
  • Local RAG
  • Codex
  • GPT-5.6

Known limitations

  • The system presents correction candidates, but it does not automatically correct the entire subtitle.
  • Full application verification requires an Apple Silicon Mac and local models.
  • The 100-case LoRA experiment is not used as the default model.
  • The current build is Ad Hoc signed and has not been notarized.
  • Paid distribution and production signing were outside the scope of this submission.
  • DeepL translation is a separate external API feature and is not included in Pocket Expert’s offline processing.

Built With

Share this project:

Updates

Submission history