LQA: Localization QA Agent

Inspiration

Machine translation made translation cheap and localization risky. A translated segment can read perfectly and still be commercially wrong, because it ignored the client's glossary. That class of bug is invisible to any reviewer who does not know the brand — which is exactly why it survives into production.

I hit this working in Romanian to English. The failure is never "the grammar is broken". It is checking account where the client's brand term is current account — fluent, natural, and wrong.

What it does

Feed it source segments, their translations, and the client glossary. It returns a scored, per-segment QA report naming exactly what is wrong and which engine found it.

It catches glossary and terminology violations, untranslated source-language leakage, probable omissions, unbalanced brackets and stray markup, duplicated words — and from the model, naturalness, register and accuracy judgements plus suggested rewrites.

How I built it

Two layers, deliberately separated.

Deterministic layer (authoritative). Glossary enforcement, source leakage, omission detection, formatting, duplicated words. Reproducible, auditable, no network, no API key. Every finding is tagged with the engine that produced it.

Model layer (advisory). NVIDIA open-source models served through Nebius Token Factory, returning structured JSON with naturalness and accuracy scores plus rewrites.

The model advises; it does not judge. A model that says "terminology OK" while a glossary term is missing does not get to overrule the checker.

The part that took the work: Romanian inflects

The glossary term is cont curent. The source says soldul contului curent. Naive substring matching reports "no issues" while the client's required term is silently missing — the tool is worse than useless, it is confidently wrong.

I implemented diacritic-folded, suffix-aware token matching: cont curent matches contului curent, while cont correctly does not match contact. That distinction is what makes it usable rather than noisy.

How it uses NVIDIA open models on Nebius

LQA's model layer is a Nebius Token Factory client (src/lqa/providers.py) speaking the OpenAI-compatible chat-completions API, targeting NVIDIA open-source models. It is selected automatically when NEBIUS_API_KEY is present, and the model, base URL and temperature are configurable.

Status: the Token Factory path is live, not aspirational. Three models were run against real Token Factory inference and the raw outputs are committed under evidence/:

The deterministic-only baseline scores 85.6. Enabling the model drops it to 66.6, and that gap is the finding: the checker can only see what it was told to look for. Nemotron-3-Ultra-550b was the only model of the three to catch a register violation — a formal vă rugăm rendered as an informal imperative — a real localization defect that no regex can detect. It also produced correct rewrites for four segments where the deterministic layer could only report that something was wrong.

An offline provider with an identical JSON contract is still shipped, so the full pipeline and the 25-test suite remain reproducible with zero credits — but the numbers above were produced live, not illustratively. The engine name appears in every report line, so there is never ambiguity about which layer produced a finding.

Challenges

• Inflection versus false positives. Prefix matching catches contului and wrongly catches contact. Solved with an explicit Romanian suffix table instead of startswith.
• Double-charging findings. Both layers independently flagged the same missing glossary term, so the score was penalised twice. Terminology is now deduplicated by expected term, with a regression test pinning it.
• Demonstrability without credits. Covered above.

Accomplishments

• 25/25 automated tests passing, Python standard library only, zero third-party dependencies
• End-to-end CLI with markdown and JSON reports and a CI gate (--fail-under, meaningful exit codes)
• Verified on planted defects: catches the inflected glossary violation, the fluent-but-wrong brand term, Romanian leakage, a dropped sentence, and a duplicated word

What I learned

That the valuable half of an AI system is often the half without the model in it. The deterministic checker is what makes the output trustworthy; the model is what makes it insightful. Shipping only the model would have produced a demo that is confidently wrong in exactly the cases that cost clients money.

Built With

• cli
• llama
• localization
• markdown
• nebius-token-factory
• nvidia
• openai-api
• python
• translation