1321

Autonomous Language Intelligence for Real-World Media. Beyond transcribe-and-translate: agents investigate real video first, then turn checked understanding into something you can watch and learn from

Comment

*NOTE: In the video, I re-ran it to show what the agent orchestration looked outside of the focus panel at around 1:14

Inspiration

Most tools treat a video like a blog with audio: transcribe, then translate. You get English words. That is not the same as understanding the video: who spoke, what the joke was aimed at, what a reference assumes you already know, or what you would need if you want to learn from the clip instead of chase subtitles.

1321 is autonomous language intelligence for real-world media. Figure out the video first, then turn checked understanding into something a person can use. Learning is a later payoff of that understanding. Korean to English on YouTube media was our focus, chosen because I am learning Korean and keep hitting that gap.

What it does

You give it media. Agents investigate it in a real workspace. An orchestrator breaks the problem into focused jobs. Specialists re-check audio and frames with tools and bring back structured evidence. The system can accept a claim, revise it, or withhold when the evidence is weak.

That understanding can be applied as private captions when they are earned, explanations on a selected line, and a scored Korean to English test on fixed hard clips where captions are a use of understanding. A separate local Codex host path runs a real investigation.

You can run the full investigation path locally. try1321.com is the product site; hosted cloud ingest is not there yet.

How we built it

Product UI: Astro and React (Studio).

Runtime: a local Node host that runs agents with clear permissions, a full audit trail of what they did, and an explicit way to withhold when evidence is thin.

Agents: Codex-directed workers with media and evidence tools.

Media tools: speech hypotheses when enabled, acoustic triage, and bounded frame sampling.

Study flow: gather evidence, cover gaps, write a study, readiness check, human approval, then private captions and QC (not public publish).

Evaluation: a frozen set of hard clips, scored by humans, with no model sitting as the judge.

Stack in short: TypeScript, Zod, ONNX Runtime, Hugging Face Transformers, Playwright for Studio checks.

How we used GPT / Codex

We used OpenAI models in two different jobs, and we kept those jobs separate on purpose.

In the product

A local host launches sandboxed codex exec processes for a root orchestrator and scoped workers. The configured model for that language work is GPT-5.6. Every child starts stripped down: read-only sandbox, no ambient shell, no web search, no apps, no memories, no built-in multi-agent. Power is removed first. Task-private MCP tools come back only when the host grants them for that exact job.

Inside that cage, GPT-5.6 decides what focused work to open next, uses the tools it was handed, and returns structured evidence. It does not own the media detectors, the journals, readiness, human approval, or private captions. Those stay host-owned. It also does not grade the benchmark. Beachhead scores use human labels with the model judge pinned null.

That design choice matters more than the model name. The hard part was not “call GPT on a video.” It was making an autonomous swarm something you can trust: grants, receipts, withhold when weak, and a clean line between investigation and publish.

In development

We also used GPT models while building: GPT-5.6 for deeper product and runtime work, and faster models like Sol when the slice was narrow and we needed turnaround. That was not “the AI coded the hackathon for us.”

The useful pattern was closer to a small engineering team than a single autocomplete box. Read-only agents mapped evidence first. One writer touched a bounded slice. Checks had to pass. Claims that the runtime could not prove were rejected. AGENTS.md and the Studio contracts stayed stricter than any model suggestion.

So the same family of models shows up twice: once as the caged cognition inside 1321, and once as a high-bandwidth collaborator for contracts, UI, evaluation, and the honesty work around them. The product only works if the second use does not quietly invent what the first use is not allowed to claim.

Challenges

The first hard problem was refusing to fake confidence. Early on it is easy to show a number you did not measure. I spent a lot of the week closing those doors i.e. null instead of zero, captures that cannot grow into scores they have not earned, withheld and unavailable as real states etc - I had to actual sit down with 2 other fluent Korean speakers to actually verify results to test progression/accuracy, especially on harder clips.

The second was building investigation under real constraints. Agents do not get ambient shell or web. They get grants, task-private tools, journals, and a path that can stop at withhold. That meant a long climb of receipted producers and an approval gate before private captions, while keeping the recorded run-006 demo and the live local host as separate authorities. If those bleed together, the UI starts lying.

The third was evaluation that can hurt the story. We froze the hard-clip pack before scoring. The first human-labeled "Bet G" result did not favor the prepared path: cold kept more critical meaning, and prep withheld more often. Later we tried one kinship instruction meant to stop inventing relatives. On the provider grid it sometimes replaced the translation with the instruction itself. Failed slots stayed failed. Incomplete pairs meant no self-improvement win. That is annoying to write about. It is also the point.

Also: macOS said my Desktop was out of space while I still had 2TB free in iCloud. The repo lived under Desktop/..., so the heavy files stayed on the SSD. iCloud quota is not local disk. And I have all of my repos under this so having to stall to wait for everything to move over was just that extra bit of "Why right now" that you remember days later.

What we learned

Real media breaks the habit of one pass and sounding confident. Studying first is not an automatic win either. On our first frozen hard-clip score, the simple cold path kept more critical meaning; the prep path withheld more often. Withholding is honest. It is not the same as being right.

If the understanding cannot eventually be reused for watching and learning, you have built a fancier converter. Agents need real tools, real evidence, and permission to say "I don't know." Labels and spawn animations without those are empty.

Built With

Share this project:

Updates