APRENDIZ

Most demos that 'learn from video' ship whatever the model says. This one audits it: our own model fabricated motion data twice, and both times the system refused it. Then it exports a Docker agent.

Comment

Inspiration

We wanted the thing everyone assumes already exists: point at a video of someone doing a task, and get back an agent that knows how to do it.

Every demo we found stopped at the interesting part. A model watches a video and produces confident prose — "push off with the hips" — and that sentence is where the demo ends and the hand-waving begins. A sentence is not a trajectory. Nobody was showing the gap between what a model says it saw and what was actually in the frame.

So we built the gap-checker first, and the product around it. The rule we set on day one was that APRENDIZ is never allowed to claim something it cannot show. That single constraint ended up shaping every design decision, and it is the reason this project has a story worth telling: our own model lied to us twice, and the system caught it both times.

What it does

You give APRENDIZ an instructional video and a sentence about what you want learned. It gives you back a Docker container you can run.

In between:

  1. Extraction. One bounded Vertex AI call turns the video into a structured, timestamped procedure — steps, evidence, and explicit uncertainties, not prose.
  2. A human approval gate. Nothing moves forward until a person reads the steps and approves them. This is not a formality; it is a hard block in the type system.
  3. Movement measurement. For physical tasks, a pose model runs locally over the video's own frames and computes joint angles from real landmark positions. Zero cloud calls, zero cost, nothing sent to a provider.
  4. Two deterministic audits. Before any of it is usable, arithmetic — not a model — decides whether the numbers could have come from a body at all.
  5. Export. An approved procedure becomes a reproducible Docker package: the skill baked in, evidence included, every provider call disabled, and a launcher that needs nothing but Docker.

What makes it different is what it refuses to do. Seven guarantees are written as Literal[False] fields on the exported skill — approved_for_execution, physically_measured, hardware_execution_approved, model_weights_updated, uploads_included, secrets_included, provider_calls_at_runtime. A package that claims more than that fails validation instead of being served. You cannot forget to check them, because a build that violates one does not build.

How we built it

Backend: FastAPI with Pydantic v2 contracts throughout. Every boundary in the system is a typed contract, and the honesty rules live in the types rather than in comments.

Extraction: Google Vertex AI with Gemini 2.5 Flash Lite, chosen after the 3.5 models returned internal provider errors on the exact YouTube path we needed.

Measurement: MediaPipe Pose Landmarker running locally over frames decoded with OpenCV. Joint angles are computed by our own arithmetic from landmark positions, with a stated flexion convention — a straight knee reads zero, not 180, and every joint carries its own neutral pose.

Infrastructure: Cloud Run, Cloud Storage for durable records, Secret Manager, Artifact Registry, ClickHouse for the audit library.

Frontend: Vanilla JavaScript, bilingual Spanish/English, no build step and no dependencies. SVG animation for both the built-in trajectory and the observed motion.

Testing: 409 unit tests and 33 browser tests, run in separate processes.

Spending: an application-enforced ceiling with an append-only ledger. We started to set up a GCP budget and then read the documentation properly: a budget alerts, it does not cap. So we built a real ceiling into the app, priced per million tokens, checked before every call and recorded after — one ledger entry per call, so concurrent Cloud Run instances cannot lose a cost in a read-modify-write race.

Challenges we ran into

The model fabricated data. Twice.

The first time, we asked for dense joint angles across twelve seconds of walking and got back 196 beautiful samples. Then we counted them. Left and right carried an identical angle in 98 of 98 paired readings. All 196 reported confidence of exactly 0.7. Each joint traced one clean triangular ramp — hip from 10° to 140° to 40° in exact 5° steps — instead of the ten gait cycles twelve seconds of walking contains. And 140° of hip flexion is anatomically impossible; walking uses about 30°.

The second time was a video of a cheetah running. 290 samples, 8 joints, confidence 0.75. Identical left and right in 145 of 145 paired readings. All 290 reported visibility clear. Every one of the 8 joints traced a single rise and fall across a window in which a running cheetah repeats its gait several times a second.

Neither response was a bug or a bad prompt. Twice, on two subjects and two kinematic chains, is a property of the method. That finding forced the biggest pivot in the project.

The interface was lying without meaning to. Our simulation console drew a built-in six-axis robot arm — correct, and always documented — but captioned it with the user's own task text under a tag reading "SOURCE / TRAJECTORY". Someone watching a cheetah video saw a robot arm labelled "how a cheetah runs" and reasonably concluded the arm was replaying their video. The panel next to it stated the truth in amber. The place people actually look said the opposite.

Tests that passed for the wrong reason. An async test using a pytest plugin we had never installed passed while never awaiting the coroutine. Two later tests encoded "this machine has no pose model" as a fact, and broke the moment we fetched one. Both were passing tests that checked nothing.

Deployment traps that no test could catch. A COPY in the Dockerfile failed because .dockerignore excluded the directory. We fixed that, and it failed again — because gcloud builds submit filters the uploaded tarball by .gcloudignore, a different file, which had already documented this exact trap after being caught by it once before. Nothing in a test suite reads the image's filesystem.

Accomplishments that we're proud of

The audit works, and we have the receipts. Two fabricated datasets, two rejections, both before the data could reach an adaptation plan or a robot. The retarget verdict says it plainly: "These samples did not survive the plausibility audit, so there is nothing here to retarget."

The Docker export is real and reproducible. Same records and same timestamp produce the same bytes and the same package id. We verified it by actually building and running one: the launcher built the image from a pinned lock file, /health answered in four seconds, the container ran as UID 10001 on a read-only filesystem with no .env, and it served the approved procedure with every guarantee false.

We measured movement without spending anything. The local pose path makes zero cloud calls and sends nothing to a provider — both permanent facts in the type, not promises in a README.

The system passed its own audit, finally. Measured samples from a real detector survived the audit that had rejected the model twice: mirrored ratio 0.0 against 1.0, hundreds of distinct confidence values against 2, zero acyclic joints against 8 of 8.

A spending cap that actually caps. Twenty soles, enforced in the application, checked before each call. It has never been hit.

What we learned

A dense answer is not a measurement. Higher frame rates bought us density, not evidence. The failure mode of a language model asked for numbers is not to refuse — it is to produce a smooth, confident, entirely invented pattern. You cannot prompt your way out of that. You can only count.

Arithmetic beats judgement for detecting fabrication. Every audit check is a number a reader can recompute from the samples themselves. The audit never asks a model whether a model was honest.

Passing an audit is not being right. This is the lesson that cost us the most and matters the most. We built a synthetic walking video where we knew the exact angles, ran the whole pipeline, and it passed both audits cleanly — while the recovered angles were wrong by double digits, with several correlations negative. The audit distinguishes fabricated from measured. It does not distinguish accurate from inaccurate. Those are two different questions, and we had only answered one.

A fixture that is its own answer key validates nothing. We could not close that gap ourselves, and recognising that is the point. Ground truth has to be authored by a person, which is why the benchmark's provenance field is Literal["person"] — a case a model labelled cannot be loaded at all.

The honest thing and the useful thing were the same thing. Every time we refused to fake a capability, we found the real one underneath was more interesting.

What's next for APRENDIZ

Ground truth first. The measurement path is built and audited but has never been scored against a human-labelled video. That is one person, one real clip, and a handful of frames marked by hand — and until it exists, we report the benchmark as unvalidated, which is not a passing grade and is never displayed as one.

Then the bridge to hardware, in the right order. We deliberately did not build the robot joint map first. Mapping a quadruped's observed joints onto a six-axis arm without an imported robot profile and a human-authored correspondence is not a conversion — it is a category error, and doing it on top of unvalidated angles would connect a robot to numbers nobody has checked.

Physics after kinematics. A simulator with real collision and dynamics checking, so "validated in simulation" means something more than a schematic projection.

And the discipline stays. Hardware execution remains impossible by type. When that changes, it will change because a safety contract was written and reviewed — not because someone flipped a boolean.

Built With

  • clickhouse
  • css3
  • docker
  • docker-compose
  • fastapi
  • google-adk
  • google-artifact-registry
  • google-cloud
  • google-cloud-build
  • google-cloud-run
  • google-gemini
  • google-secret-manager
  • google-vertex-ai
  • html5
  • javascript
  • mediapipe
  • opencv
  • playwright
  • pydantic
  • pytest
  • python
  • python-multipart
  • rest-api
  • svg
  • uvicorn
Share this project:

Updates

Submission history