Inspiration

In robotics the human is usually the teacher: robots learn from imitation, teleoperation and motion capture. We ran that pipeline backwards for the hackathon challenge "Flip the Script". A simulated robot shows you a pose, watches you copy it through a webcam, and tells you joint by joint what to fix.

What it does

• Pollen AmazingHand teaches British Sign Language handshapes.
  • It teaches ONE, TWO, THREE, FIVE, SEVEN and SORRY. SORRY also checks the circle your wrist draws. This will be extended to teach the whole of BSL handshapes.
  • You can type a sentence, such as "Good morning, I'm sorry". The robot teaches each sign, signs the whole statement, then watches you sign it back.
  • It marks your fingers green or red and gives corrections like "Index: curl 20° more". You hold the shape to pass.
  • Words with no sign are listed on screen and skipped, never guessed.
• Unitree G1 humanoid teaches martial arts stances, limb by limb.
  • It teaches seven stances: Ready, Horse, Bow, Fighting guard, Crane, White crane spreads its wings, and Archer. Each can be copied like a mirror or on the same side.
  • It scores each limb and gives one correction.
  • An optional AI sensei suggests the next stance and says one sentence after each rep.

How we built it

• Hand:
  • MediaPipe hand landmarks are reduced to 8 numbers: curl and spread for the thumb, index, middle and ring fingers. The AmazingHand has no little finger.
  • The robot hand is described in the same 8 numbers, so the two compare per finger.
  • Curl is measured on MediaPipe's 3D world landmarks, so it doesn't depend on camera distance.
  • Calibration defaults were measured on a real laptop webcam, and c recalibrates to your own hand.
  • Every sign comes from a written description in a public BSL dictionary, cited per sign.
• Humanoid:
  • MediaPipe Pose and the G1 in MuJoCo are both turned into 8 limb directions in their own torso frame. Directions ignore limb length, so the robot's proportions don't matter.
  • A limb counts as right within 30°.
  • A k-nearest-neighbour recogniser, trained on recorded examples and evaluated leave-one-person-out, names the stance you're actually in.
• Optional AI on Nebius Token Factory, each with an offline fallback:
  • Qwen3-Embedding-8B picks a move from a request in words.
  • Qwen3-235B-A22B-Instruct-2507 acts as the sensei.
  • MiniCPM-V-4_5 gives a vision tip. It is off by default.
  • Nebius is called once per request or per rep, never per frame.
• Pitch video: made with Remotion, with voiceover by OpenAI gpt-audio and music by Google Lyria 3.
• Privacy: webcam frames stay in memory. The exceptions are opt-in: rth record saves labelled recordings for training, and the vision tip sends one downscaled snapshot when switched on.

Challenges we ran into

• The AmazingHand has four fingers and one hand, so many signs are approximations. Each sign is marked exact or approximated.
• Two-handed signs and fingerspelling had to be left out.
• It checks handshape only. It doesn't check location, palm direction or facial expression, which are part of BSL.
• Everything was built in one day.

Accomplishments that we're proud of

• Two working robot teachers, a signing hand and a martial-arts humanoid, built in one day.
• Typed sentences become a lesson, with the robot teaching and then checking each sign.
• The humanoid's feedback is per limb, and the stance recogniser was evaluated leave-one-person-out.
• The booth works with no API key because every AI feature has an offline fallback.
• Webcam frames stay in memory, with only opt-in exceptions.
• Every sign cites its source, and the app says plainly what it can't check.
• The suites need no camera or network, apart from a few vision tests that use downloaded models.

What we learned

• Describing human and robot bodies in the same few numbers makes comparison and correction simple. Using limb directions rather than positions removes the robot's proportions from the problem.
• Calibrating on a real webcam matters.
• Being explicit about what the system doesn't check keeps the claims honest. It is not an accredited way to learn BSL, and no fluent signer has reviewed the set.

What's next for RobotTeachesHuman

Picture a kitchen table late in the evening. A parent whose child has just been diagnosed as deaf has a BSL lesson on Thursday and nobody to ask until then. A robot hand sits on the table and shows a sign, slowly. The parent copies it. One finger turns red: "Ring finger, curl 20 degrees more." They fix it, and it turns green. That moment is what we built in six hours.

Next we want it to be true for more people. First, trust: fluent BSL signers will check every sign, and we will test the marking on many different hands, so green and red mean what they say. Then more of the language: palm direction, movement, and eventually both hands and the face. Then the room itself: we will build the real, open-source Amazing Hand, so the teacher sits beside you instead of on a screen, and add a spoken tutor, grounded in our sourced signs, so you can ask why.

The same loop can teach any movement. We have recorded robot movements for yoga and karate, and with clinicians, it could one day guide hand exercises at home. For years we taught robots to move like us. Next, they help us learn.

Built With

• claude
• mediapipe
• mujoco
• openai
• python
• typescript