Copy Review

Copy Review uses GitLab CI and Duo to catch multilingual UI defects, test proposed fixes, and follow reviews through to resolution.

Comment

Inspiration

Localization opens a product to more people. Facebook's growth team identified internationalization as one of its major early growth drivers, alongside helping users connect with friends. In everyday business, Stripe uses customer language preferences to localize invoice emails and PDFs. (Facebook growth account, Stripe invoicing documentation)

Shipping a translation is only the beginning. Every release can change a label, a layout or what a message promises. A feature can work technically while a translated instruction hides an option or gives a customer the wrong expectation. Developers, testers and language reviewers still have to find the defect, explain it, coordinate a correction and check the next revision.

Our controlled invoice demonstration makes that recurring work visible. A German search hint needed 445 pixels in a 195-pixel mobile field, hiding the invoice-number option. An Indonesian notice promised email delivery even though the app only simulated sending. After a person corrected both translations, CI passed and Duo followed up on the original findings.

Copy Review is for teams shipping multilingual software. It brings screen evidence, wording review and correction follow-up into GitLab, so the review continues after a finding is posted.

What it does

Copy Review combines a GitLab CI capture job with a custom GitLab Duo Agent Platform flow. A completed MR pipeline starts the review automatically.

  1. Render and measure. CI selects affected registered Storybook screens, renders them in seven languages at desktop and mobile sizes, and checks translated elements for overflow, missing translations, raw keys, broken placeholders, and instruction-like content.
  2. Review meaning in context. Duo checks the report's project, pipeline, job and commit, then reviews changed strings alongside their source text, nearby labels and repository language policies. It looks for meaning drift, inconsistent terms, mixed formality and misleading literal wording.
  3. Propose and test corrections. For selected fit and placeholder findings, the released v1.2.0 flow tests proposed replacements against the exact captured app snapshot before offering an Apply suggestion. Mechanical verification does not approve translation meaning.
  4. Route findings. Developer findings get inline MR suggestions. Language findings become checklist items in a standing issue for that language. Missing language policies can become a proposed MR for human review.
  5. Follow the next revision. People apply corrections. CI runs again, and Duo updates the original findings and checklist instead of leaving a disconnected report behind.

Proven checks can fail CI. Language judgments remain advisory. People control application changes, policy approval and merging.

Autonomy and human approval

Category: Start Fresh path, Supervised autonomy. The review steps run automatically, including publication of findings and policy proposals. People assess wording, apply application fixes, approve policies and control merging. The flow does not merge or deploy changes. Mechanical checks can fail CI; language judgments remain advisory.

How we built it

The invoice app uses React, TypeScript, i18next, Express and SQLite. Edits and simulated-send activity persist through reload. It supports English, Indonesian, German, Norwegian Bokmål, Korean, Simplified Chinese and Arabic.

A Node.js runner uses Playwright and headless Chromium to capture Storybook screens. A manifest maps translation keys and source files to stories; changes outside that mapping trigger broader capture. Duo checks that each report belongs to the correct project, pipeline, job and commit. A dedicated service account publishes the review through GitLab tools.

The candidate verifier runs in a container and checks replacement text, placeholders and surrounding elements against the captured app. Teams must register their screens, configure CI and language policies, and enable the Duo flow. Coverage is limited to registered screens and captured states.

Workflow architecture

Copy Review workflow architecture: CI capture, evidence validation, Duo language review, candidate verification, GitLab findings, human correction and follow-up

See it work

  • MR !12: clipped text and an incorrect email promise. The historical v1.1.1 run produced a German inline suggestion and an Indonesian language issue. After both strings were corrected, CI passed and Duo followed up. One optional style advisory remained.
  • MR !16: verified suggestions. In v1.2.0, Duo verified a shorter German label and a restored Indonesian placeholder before publishing them. A person applied both and requested a full recapture across 280 story/language/viewport combinations. CI passed, and Duo resolved the original and recovery threads. Both candidates passed on their first attempt.

We also exercised the flow on new feature copy, a simulated translation delivery and missing-language policy proposals. The 2:58 video shows the recorded workflow and labels the separate runs.

Challenges

A plausible suggestion is not enough. It has to target the right line and revision, preserve placeholders and fit the actual screen. We had to get the verifier working inside Duo's runtime, reproduce the original defect, and check the published output against the capture. Missing evidence must produce an incomplete result, not a pass.

Fitting on screen also does not make a translation correct. We kept layout checks separate from wording review. The verifier can request one revised candidate, but successful second-attempt revision has only been tested locally with supplied candidates, not demonstrated live with Duo.

What we learned

In an earlier test with 28 planted problems, Copy Review scored 25/28 under a strict category-matching rubric; a general-purpose agent given the same evidence scored 28/28. Copy Review detected all 14 mechanical problems. Its three misses were wording problems it found but assigned to a different category.

That test did not show better language judgment. Our contribution is connecting the review to GitLab: tested suggestions, language issues and automatic follow-up on the next revision.

Eight fresh changes: observed results

We then ran v1.2.0 on eight new synthetic Indonesian changes in the same invoice app:

  • 8/8 captures and Duo reviews completed. Quality and UI-verifier jobs passed on all eight branches; that does not mean every translation was correct.
  • Both published placeholder fixes passed a separate render check, covering ten story/viewport combinations against their exact CI snapshots.
  • Four language findings were confirmed in GitLab, attached to the correct MR and source revision.

Six issues flagged, six backed up. An agent audit supported every issue Duo reported and agreed the two changes it left alone were clean. These were synthetic changes in a familiar app; unfamiliar repositories with fluent reviewers come next.

The median time from pipeline creation to the first review was 9m34s (range 4m08s to 13m18s), including queueing from launching all eight trials together. This is elapsed automation time, not reviewer effort or time saved.

In a separate agent-operated replay of MR !16 with the same evidence, the flow supplied both replacement strings, started verification itself and resolved both original findings, which the operator otherwise did by hand. That shows which tasks moved to the flow, not minutes saved.

Estimated time savings

Our illustrative model uses 8 minutes for manual review → 5 minutes with Copy Review: 3 minutes saved per localization review.

Localization reviews Estimated reviewer time saved
10 30 minutes
100 5 hours
1,000 50 hours

These are projections, not measured savings or tested processing volumes. Totals are before setup and maintenance.

The video's scenario adds those costs: 60 minutes of setup and 15 minutes of maintenance a week. At 10 reviews a week, 30 minutes saved minus 15 minutes of maintenance leaves 15 minutes a week, so setup pays back in four weeks. At 5 reviews a week, maintenance cancels the saving.

What's next

Test on unfamiliar projects with fluent reviewers, measure active review time, and make screen registration easier. We also want to improve handling of concurrent reviews and fixes that restore earlier strings.

Built With

Share this project:

Updates

Submission history