-
ui
-
main menu
-
ui-1
Inspiration
LLM Insufficiency for natural language processing-based task completion in localization workflows.
What it does
ToFU (text-over-frame-unification) is a seven-layer visual translation pipeline that detects, erases, and re-renders text in images while preserving the original scene's visual context. Given a source image and a target language, ToFU runs text detection and recognition (Cicerone), semantic surface analysis (Scene), source-text erasure and background inpainting (Cleanse), style-matched target-language rendering (Scribe), and quality verification (Verify) — with a visual translation memory (Memory) that accumulates approved localizations for reuse on future assets. Three auxiliary modules — Savor, Menu, and Wasabi — operate as post-recognition quality-control passes inside Cicerone, correcting glyph-level confusions, recovering known place names, and normalizing CJK character variants. The pipeline is engine-agnostic (EasyOCR or PaddleOCR), runs entirely on CPU, and gates output on measurable accuracy thresholds.
How I built it
Computer Vision / Image Processing Pillow (PIL) – image I/O, rendering, font glyph bitmaps OpenCV (cv2) – opencv-python-headless for scene/contour/geometry work NumPy – image array interchange scikit-image – threshold_sauvola in tofu/utils/imaging.py scipy – transitive backbone for scikit-image OCR / Text Detection EasyOCR – default Latin-strong detector/recognizer (CRAFT + CRNN) PaddleOCR / PaddlePaddle – CJK engine (PP-OCRv5), runs in an isolated interpreter because it force-replaces numpy/OpenCV torch / torchvision – EasyOCR backbone (CPU build in app venv) Inpainting & Segmentation (optional, isolated venvs) simple-lama-inpainting – neural background repair (.venv-inpaint, CUDA torch) segment-anything (SAM) – alternative scene backend in tofu/layers/scene.py NLP / Linguistic Processing stanza – basil.py linguistic enrichment (local models only, gated by TOFU_BASIL_STANZA_DIR) Font / Text Shaping fonttools – glyph coverage, font introspection uharfbuzz – HarfBuzz shaping for Indic/SE-Asian scripts freetype-py – FreeType glyph bitmap rendering arabic-reshaper – RTL contextual joining forms python-bidi – UAX #9 logical→visual reordering Web Backend / API FastAPI uvicorn (standard extras) pydantic v2 python-multipart Frontend (React/TypeScript) react / react-dom v19 @mui/material / @emotion/react / @emotion/styled lucide-react / react-icons ogl – WebGL tailwindcss / @tailwindcss/postcss vite / @vitejs/plugin-react typescript vitest Data / Glossary / Utilities openpyxl – .xlsx glossary ingest xlrd – .xls glossary ingest gradio – Hugging Face Space playground only Dev / Build pytest ruff / mypy (configured in pyproject.toml)
What's next for ToFU (text-over-frames unification)
improved OCR for video localization
