Mnema

The translation workspace with a memory. Context-aware localization for film and comics, built to keep an entire story consistent.

Comment

Inspiration

I used to work as a book translation editor, and that job taught me something that stuck with me: translation is never really about knowing two languages. A good translation carries the things that live around the words. The context, what the author was trying to do, the tone of the text, and all the small editorial decisions made earlier in the book that have to hold together to the very end.

Even strong translators and editors, people who know the source language inside out, miss things in a long and tangled text. A recurring idea. A quiet shift in perspective. A piece of context that was set up much earlier. When that happens you get "mechanical" translation, and over time I came to see that this is not only an AI problem. It is a workflow problem. The context and the memory are already there in the work. They are just not visible enough when someone is heads-down on a single line.

Human translators build that depth the slow way, through experience, notes, revision after revision, and an editorial instinct that is hard to put into words. The thought behind Mnema was that part of this process could be carried by the system itself.

So Mnema started from one idea: translation should not be treated as a one-time text generation task, but as a living localization process shaped by evidence, context, memory, and human review.

What it does

Mnema is a translation workspace with a memory underneath it. Instead of treating a line of dialogue as a string to be swapped from one language to another, it keeps track of everything that gives that line meaning: what is on screen when it is spoken, who is speaking and how they talk, where the story has been, and the tone it is carrying. That memory sits behind every suggestion it makes. A translator is never starting from a blank line. They are working with the same full picture a human translator would have if they could hold an entire film in their head at once.

The platform builds its workflow around the project itself. A film and a comic do not get localized the same way, so Mnema gives each its own workspace with the steps that fit the medium.

In the Movie workspace, Mnema lines the subtitle file up against the video as soon as both are uploaded. Every cue is matched to the frames it plays over, and from those frames it reads the scene. I call this the visual context. So before a single word is translated, the line already knows what it is sitting on top of: the shot, the moment, and the mood it belongs to.

Translating one line then pulls on far more than that line alone. Mnema gathers a full context bundle around it:

  • the source line together with the cues right before and after it, so dialogue is read as a conversation instead of a list of separate sentences
  • the visual context taken from the frames the line maps to
  • the memory it has built up: glossary and terminology, how each character speaks, the tone between characters, facts the plot has already established, continuity notes tied to what is visible on screen, and any line that has already been approved earlier in the project
  • a running account of the story so far: a summary of the project, the scene in play, where the plot currently stands, and the state each character is in

All of it goes to the model at the same time, so the translation it comes back with is weighed against the same things a careful human would consider, not just the words on the line.

The interesting part is how Mnema decides which pieces of that memory are relevant in the first place. Everything is stored in Aurora PostgreSQL, and Mnema remembers in two ways at once. Some memory is matched by rule: a locked glossary term or an already approved translation should win every time, with no guessing. The rest is matched by meaning. Using pgvector inside Aurora, Mnema compares the current line against everything it has stored and finds the moments that are close in meaning, even when they share none of the same words. The two results are merged, so each translation is grounded in both the hard rules the team has set and the softer, more human sense of "I have seen something like this before."

Nothing gets used until a person signs off on it. Every suggestion Mnema makes is a proposal, not a result. The translator reads it and either approves it as the line that ships or corrects it. The correction is where the system gets smarter. When someone fixes a translation and notes what was wrong with it, Mnema turns that into a lesson it can reuse, and the team can lock that lesson so it becomes a hard rule the model has to respect from then on. An approved line does not just vanish into the file either. It becomes memory, so the next time something similar comes up, the work already done is right there shaping it.

Once every line in the project has been approved, Mnema makes one final pass. A stronger model reads back over the whole set of approved subtitles together, the way an editor reads a finished draft from the top, and flags anything that still feels off: a term that drifted, a tone that slipped between scenes. These are suggestions, not changes forced on the work. Each one can be accepted and edited or left as is, and none of it can block delivery. It is a second pair of eyes, not a gate.

When the work is done, Mnema hands back something that is genuinely ready to use. Timing can be nudged with a simple offset so the lines sit exactly where they belong against the video, and the project exports as a finished subtitle package, ready to download and ship.

Because every kind of work has its own shape, no single workflow could ever cover all of them. A comic is not watched along a timeline. It is read panel by panel, balloon by balloon, and the layout itself is part of the meaning. So the Comic workspace runs on the same engine underneath, the same memory, the same context bundle, the same approve-correct-learn loop, but the surface is built for the page instead of the timeline.

A comic comes in as a series, broken into chapters and pages. Instead of matching cues to frames, Mnema reads the page directly. It uses Amazon Textract to find the text, works out where each speech balloon and caption sits, and puts them in the right reading order. Every region gets its own box on the page, tagged for what it is, whether dialogue, narration, a sound effect, or a sign, and any of it can be adjusted: fix the text, move the box, mark a region in or out of the final export, or note who is speaking.

And just like on film, Mnema keeps a running summary of the story as it goes, page by page and chapter by chapter, so a name, a running joke, or a plot point set up early stays consistent all the way through. From there the path is the same: Mnema proposes a translation for each region with the full context behind it, a person approves or corrects it, and the lessons feed back into memory. When the chapter is done, Mnema renders the translated pages with the new text laid back into the balloons, so what comes out is a finished localized page, not just a list of strings.

Across both workspaces it works the same way: Mnema does the remembering and the groundwork, and a person makes the call. Translating one good line was never the hard part. Keeping a whole film or series consistent, from the first cue to the last, is what Mnema is for.

How we built it

Mnema is a Next.js app running on Vercel, and it leans on a small set of AWS services that each do one job well. The interface and all of its actions run on Vercel through the App Router and server actions, so there is no separate API tier to keep in sync. The UI itself started from a v0 theme that I took as a base and reshaped around the workspace. The data, and the thinking that depends on that data, live on AWS.

Aurora PostgreSQL is the spine of the whole thing. Every project, every line, every approved translation, every piece of memory, and every audit record sits there as the single source of truth. The schema is deliberate rather than a catch-all: memory is typed and scoped, so a glossary term, a character's voice, a continuity note, and an approved line are each stored as a distinct kind of memory with its own rules, and each one is attached to the right level, whether that is the organization, the project, or a single asset.

The part I am happiest with is how that memory gets retrieved, and it is where Aurora really earns its place. On top of Postgres I use pgvector to store an embedding for every piece of memory, generated through Amazon Bedrock's Titan model. When Mnema needs context for a line, it runs a vector search inside Aurora to find the memory that is closest in meaning, then merges that with the rule-based matches, the locked terms and approved lines that should always win. It is a single query doing two kinds of remembering at once, with no separate vector database to keep running on the side. Embeddings are cached by content hash, so the same one never gets computed or paid for twice. And when two pieces of memory disagree, an older translation against a newer decision, Mnema surfaces the conflict and lets the team resolve it by keeping one, replacing it, or splitting it into separate cases, instead of guessing.

All of the language work goes through Amazon Bedrock, and each task is routed to the model that fits it: Nova Pro for the heavy narrative reasoning across a full timeline, a faster Claude model for quick passes, a stronger one for the final proofread, and Titan for the embeddings. Routing is set per project, and because every model call goes through one place, I can price exactly what each one costs.

Some work simply does not belong on Vercel, so it does not run there. Pulling frames out of a video with ffmpeg is heavy, so that happens on an ECS Fargate worker instead. The browser uploads media straight into a private S3 bucket through presigned URLs, with resumable multipart uploads for large video and checks on content type and size before anything is accepted. When the worker finishes, it reports back through an idempotent callback, so a retried or duplicated message never doubles the work. Comic pages go through Amazon Textract the same way, and everything, source files, extracted frames, and exports, stays private behind signed URLs.

There are no AWS keys anywhere in the project. Vercel's OIDC identity assumes a least-privilege IAM role at runtime, scoped to only the S3, Bedrock, and Textract actions Mnema needs. The entire AWS side, the role and the Fargate stack, is defined in CDK, so the infrastructure is versioned right alongside the code.

On top of that sits the part that makes this a product rather than a demo. Sign-in and sessions run through Better Auth, and every server action checks two things before it does anything: that the caller is allowed to take that action, and that the project they are touching belongs to their organization. That isolation is enforced on the server, on every call, never assumed in the interface. Behind it is a real credit ledger: each organization has a balance, every AI call is priced by the model it used and written to the ledger with an idempotency key, and a usage dashboard shows where the credits went. A public read-only role lets anyone open the workspace and look around without being able to change anything, which is how the demo stays open without being fragile.

The rest is the unglamorous reliability work that keeps it honest. Long-running jobs take an operation lock so the same translation or export cannot be started twice at once. Candidates pass through a quality gate before they are ever offered. If the database is not configured, the app drops into a clearly labeled read-only mode instead of crashing. And the whole thing is held together by 85 tests, an end-to-end suite, and a single check that runs linting, the tests, the production build, the database schema check, and the infrastructure synth before anything ships.

Challenges we ran into

The hardest problem was not a technical one. It was trying to take what a good translator does in their head and turn it into something a system could follow. A skilled translator carries an enormous amount of context without ever spelling it out, and almost none of it is written down anywhere. Working out which parts of that matter for a given line, and just as importantly which parts are only noise, took a lot of careful thinking before I wrote much code at all. My first instinct was to remember everything, and that turned out to be its own kind of failure. Too much memory buries the model in detail and the translation gets worse, not better. A real share of the work went into deciding what to keep, what to leave out, and how to rank whatever was left.

Closely tied to that was keeping Mnema in its lane. It was surprisingly easy to drift toward building a translation agent, something that just produces an answer and moves on. That was never the point. I wanted an assistant that supports a localization team without taking the decision away from them, and getting that balance right meant reworking the workflow and the prompts over and over, so the system proposes a translation, explains why, and then waits for a person, instead of quietly deciding on its own.

The comic side brought a much more physical problem. Finding the text on a page is one thing; putting a translation back onto it cleanly is another. Detecting the balloons and reading them in the right order took real effort, and laying translated text back into them so the page still looks intentional was harder still. I will be honest about where this landed: dialogue balloons work well, but sound effects are still unsolved. SFX are drawn into the art itself, stylized and stretched and part of the illustration, and I do not yet have a way to render a translated version of them that is clean enough to publish. For now I have left that as a visible edge rather than faking it.

A few of the challenges were lower level but just as stubborn. Tuning the memory retrieval was a constant balancing act: search too loosely and the model gets context that has nothing to do with the line, search too tightly and it misses the one connection that mattered. Finding the right thresholds for the vector search, and getting the rule-based and meaning-based results to merge without contradicting each other, took a lot of trial and error against real dialogue.

Keeping the heavy work off Vercel created its own set of problems. Frame extraction is far too slow to sit in a serverless function, so it had to move to a separate worker, and the moment work happens somewhere else you inherit all the usual distributed headaches: messages that arrive twice, jobs that get retried, results that come back late. Making the callback idempotent, so a duplicated message never doubles the work or double-charges the credit ledger, was one of those things that sounds trivial and quietly eats a day. Wiring Vercel's OIDC identity to a least-privilege IAM role, so the app could reach S3, Bedrock, and Textract without storing a single key, was the right call for something meant to ship, but scoping the trust and permissions correctly took its share of patience too.

Accomplishments that we're proud of

The thing I am proudest of is also the easiest to see: the translations get visibly better once the memory is doing its job. You can run the same line with the memory turned off and then with it on, and the difference is not subtle. A character keeps their voice, a term that was settled fifty lines earlier stays the same, a callback that only works if you remember an earlier scene actually lands. That side by side was the moment the whole idea stopped being a theory for me. It is one thing to argue that context matters; it is another to watch a translation improve in front of you because the system remembered something a person would have.

Getting the hybrid retrieval to actually work is the engine behind that. Combining strict, rule-based memory with meaning-based search in pgvector sounds clean on paper, but making the two cooperate, so the locked rules always hold while the vector search fills in that human "I have seen this before" instinct, took real work. When it finally clicked, it stopped feeling like a database query and started feeling like recall.

I am also proud that Mnema does not stop at a good suggestion. It carries the work all the way to a finished deliverable. Once every line is approved and the final proofreading pass is done, you can adjust the timing and export a real subtitle package, or render translated comic pages with the text set back into the balloons. Plenty of tools can produce a translation; far fewer hand you something you could ship at the end. Closing that last gap, from "here is a draft" to "here is the file," mattered to me.

And underneath all of it, this turned into a real piece of software rather than a weekend demo. Proper sign-in and sessions, organizations that are isolated from each other on the server, a working credit ledger that prices every model call, infrastructure defined in code with no stored keys, and a test suite and build that have to pass before anything ships. Building that across two completely different media types, and having it hold together, is the part I quietly did not expect to finish.

What we learned

Most of my earlier AI work lived on the gaming side, or on the more theoretical end of model consistency, the kind of problem you can explore without anyone ever having to use the result. This was the first time I set out to build something B2B, an actual product meant for a real team to depend on, and that changes almost every decision. The question stops being "can the model do this" and becomes "will a localization team trust it, fit it into the way they already work, and get something they can ship out the other end." Designing around someone else's workflow, instead of around an interesting problem, was a genuinely new muscle for me.

It was also the first time I leaned on pgvector and Amazon Aurora this heavily, and what surprised me was how much I did not have to add. I went in half-expecting to bolt a separate vector database onto everything, and instead the relational data and the semantic search ended up living in the same Postgres, queried side by side. Keeping the entire memory layer, structured records and meaning-based search alike, inside one Aurora database made the whole thing far easier to reason about than I expected, and it is a big part of why one person could keep all of it in their head at once.

The lesson that stuck hardest, though, was that more memory is not better memory. My instinct was to hand the model everything and trust it to sort things out, and that reliably made the translations worse, not better. The value turned out to be in restraint, in picking the few pieces of context that mattered for a line and leaving the rest out. Building Mnema taught me that the hard part of a memory system is not storing things. It is deciding what to forget.

And somewhere along the way I learned what "shippable" really costs. The interesting model work is maybe half of it; the other half is the unglamorous trust-building, the sign-in, the isolation between teams, the metering, the infrastructure that has to be right before anyone would put real work into it. That part is invisible when it works, and it is exactly the part that separates a demo from something a team would adopt.

What's next for Mnema

The most obvious direction is opening up the tracks that are still locked. Film and comics were the right place to start, but each new medium brings its own shape, and the ones I am most drawn to are the ones that demand a far larger sense of context. A TV series is really a memory problem stretched across episodes and seasons, where a character's voice, a running thread, or a detail planted in an early episode has to survive dozens of hours of story. Books push that further still: hundreds of pages where the narrative voice and the choices made in chapter one still have to hold in the final chapter, with room for things like translator's notes. Games are a different beast again, with non-linear and branching text, interface strings, variables and placeholders, and hard limits on length the translation has to respect. Each of these needs its own workflow and its own set of features rather than a reskin of the movie one, and that is exactly the kind of problem Mnema's memory layer was built for.

Next is the quality of what comes out. I want the exports to be not just correct but finished, single clean deliverables with nothing left to tidy up by hand, and I want to give people more professional editing tools inside the platform itself, so refining the final output is something they do in Mnema rather than exporting it and fixing it somewhere else. The honest gap from the comic side belongs here too: I still want to crack rendering translated sound effects cleanly enough to publish, which is the one place the page output is not there yet.

I also want to go deeper on language itself. Right now the memory mostly understands a specific project; the next step is memory that understands a language. Building per-language context, the way a language expresses things, alongside slang and idiom dictionaries, would let Mnema share a common layer of knowledge across every project in that language, so it gets better at, say, English to Turkish the more work passes through it, instead of only getting better at one film.

And a couple of things fall out naturally from where the product already is. Because organizations and roles are already in place, real team collaboration is a short step away: assigning lines or chapters to different reviewers, review queues, and several people working the same project at once. Beyond that, I want memory to compound across projects, so a glossary or a lesson locked on one job carries into the next, and a studio's accumulated decisions become an asset that grows with every title it localizes.

That is the version of Mnema I am building toward: not a tool you reach for one line at a time, but a memory a localization team builds once and keeps.

Built With

Share this project:

Updates