Sword

A sword-fighting robot using a Quest 3. The headset's passthrough camera is the only sensor; an xArm 6 fences back in real time.

How it works

Capture. Quest Browser reads the passthrough camera track directly through MediaStreamTrackProcessor with a one-frame queue, encodes the newest frame to JPEG, and streams it over a WebSocket to a local server. Each frame answers a server-issued challenge, so a sample can never be older than the request that asked for it. Anything late, out of order, or ambiguous is dropped rather than interpolated.

Vision. OpenCV detects an ArUco marker on each sword and a printed checkerboard fixed beside the robot. Poses come from IPPE PnP against a calibrated lens model, with reprojection and planar-ambiguity checks before a pose is accepted. The human sword is expressed in the board's frame, which cancels headset motion:

$$ T_{\text{sword}}^{\text{board}} = \left(T_{\text{board}}^{\text{cam}}\right)^{-1} T_{\text{sword}}^{\text{cam}} $$

Control. You hold a neutral guard once. From then on the arm's target is the guard pose plus the scaled change from neutral:

$$ p = p_{\text{home}} + k_p\,M\,(p_{\text{sword}} - p_{\text{neutral}}), \qquad R = \exp!\big(k_r \log(M\,R_{\text{sword}}R_{\text{neutral}}^{\top}M^{\top})\big)\,R_{\text{home}} $$

where \(M\) maps board axes into robot axes. The target goes through the controller's native inverse kinematics, joint aliases are unwrapped relative to the measured pose, and the result is projected into a bounded workspace. A velocity loop with acceleration and stopping-distance limits drives all six joints.

Challenges

  • Latency on Quest. The async canvas.toBlob path stalled for 180–940 ms per frame on the headset. Encoding synchronously brought it to ~26 ms.
  • Planar pose ambiguity. A single 50 mm marker flips between two valid orientations when viewed near head-on. A larger fixed checkerboard as the world reference, plus a short-lived orientation prior transported through the board, resolved most of it.
  • Firmware quirks. The xArm's IK can't be seeded, latches a warning on no-solution queries, and returns a slightly different commanded position than the encoders report. Each needed explicit handling instead of a workaround.
  • Clock alignment. Camera timestamps and performance.now() have different origins on Quest; they're aligned by the minimum observed delivery offset and requalified after any gap.

Built With

  • metaquest3
  • xarm
Share this project:

Updates

Submission history