Inspiration
Robot foundation models are not short on architecture. They are short on demonstrations. Today every demonstration is recorded by hand: data factories full of teleoperators in VR headsets, workers paid by the hour to wear motion-capture suits, gig platforms that pay people to do chores for a camera. Meanwhile TikTok and YouTube already hold billions of videos of people doing every task we want robots to learn.
We wanted a brain that closes that gap: tell it what you want a robot to learn, and it goes and finds the people who already did it on camera, turns their motion into robot training data, and gets smarter about where to look every time it runs.
What it does
You type one sentence, like "train a Unitree G1 to do a dumbbell shoulder press". An AWS Strands agent then runs four tools:
- Recall (Cognee): what did past runs learn about this task and this robot? Videos that already failed are skipped.
- Discover (Bright Data): search the open web for videos of one person clearly doing the task. Every lead must pass a Creative Commons licence check.
- Build: download, decode each untrusted video inside a Docker sandbox, track 33 body points per frame with MediaPipe, judge the footage, and retarget the motion onto a real robot model in MuJoCo physics.
- Remember (Cognee): store every outcome and one lesson from the run, so the next run starts smarter.
Every episode keeps its source, licence and author. In one recorded run the funnel was
$$68 \text{ found} \;\rightarrow\; 34 \text{ judged} \;\rightarrow\; 22 \text{ accepted demonstrations}$$
with no human in the loop.
How we built it
We started from Player Two, which we built two days earlier at The Executable World hackathon (video in, robot demonstrations out), and added the brain around it.
- Brain: a Strands agent in Python with four
@toolfunctions. The model plans the searches and the order; it never judges footage. If the model is unreachable, the same four steps run in a fixed order. - Memory: each video outcome is written to Cognee as one structured line (video, task, robot, outcome, reason) so recall can parse it back exactly, plus a free-text lesson per run for graph search.
- Sandbox: downloaded video is copied into the one folder a Docker sandbox can see, decoded and pose-tracked in there, and only a JSON file of numbers comes back out.
- Judge: deterministic checks on framing, one continuous person, tracked frames, and whether the planned motion actually happens in the planned direction. Each episode gets a quality score
$$q = 0.6\, s_{\text{judge}} + 0.4 \cdot \frac{t}{100}\left(1 - \frac{c}{100}\right)$$
where $t$ is the percentage of control ticks where the person was tracked and $c$ the percentage spent at the robot's speed limit. Episodes with $t < 85$ are dropped.
- Retargeting: numeric inverse kinematics on each robot's real kinematics (Unitree G1, SO-101, Franka Panda), solved with Levenberg-Marquardt steps
$$\Delta \mathbf{q} = \left(J^\top J + \lambda I\right)^{-1} J^\top \mathbf{e}$$
inside MuJoCo compiled to WebAssembly, so the same simulation runs in the browser and on the server.
Challenges we ran into
- Cognee and Bright Data never went live. Cognee rejected the API key we had, and our Bright Data account was suspended pending billing. Both integrations are complete, and the agent was built to keep running without them, but on demo day memory and web discovery did not contribute to the recorded runs.
- One camera has no depth. MediaPipe guesses that a standing person's knees point toward the camera, which made the robot crouch. We switched to estimating leg depth from foreshortening instead.
- The LLM described a shoulder press as a "forward" motion, so the judge rejected eight good clips. Now the planner cannot claim a direction the task's own words don't support, which took that run from 0 to 6 accepted.
- Tasks that make no sense for the robot. Asking a one-arm SO-101 to do jumping jacks is refused before anything is searched, with a suggestion to switch robots.
- Strands without AWS credentials. There were none on our machine, so the Strands agent runs on the Gemini model provider instead of Bedrock.
What we learned
- Quality gates beat volume. Most video is rejected, and that is the product.
- Let the LLM plan, not judge. Every accept or reject is a measured, repeatable check.
- Memory changes an agent's economics. Skipping downloads that already failed for the same task and robot saves the most expensive step.
What's next
Get the live Cognee and Bright Data runs going, add hand tracking so the gripper can close, and train a first policy on the demonstrations the brain collects.
Log in or sign up for Devpost to join the conversation.