Sole Mates: Sock Matchmaker with BracketBot

Find matching socks, plan a two-arm reunion, validate every Cartesian motion against the supplied BracketBot URDF, and escorts unmatched socks to the Singles Club.

Comment

Inspiration

Everyone has a drawer full of unmatched socks. It's one of life's small frustrations and it turns out it's also a genuinely hard robotics problem. Matching socks requires vision (color, pattern, shape), planning (which arm picks which sock), and coordination (two arms moving simultaneously to bring a pair together). We wanted to build something that was simultaneously fun, technically deep, and demonstrably complete within a hackathon.

What it does

Sole Mates takes an overhead camera image of separated socks on a table and:

  1. Detects every sock using OpenCV background segmentation and morphological cleaning
  2. Characterizes each sock with an HSV color histogram, Laplacian texture statistics, PCA orientation, and shape descriptors (compactness, extent, aspect ratio)
  3. Matches pairs using a weighted similarity score and dynamic programming to find the globally optimal non-overlapping pairing (not just greedy best-first)
  4. Plans grasp poses for each sock using distance-transform interior points and PCA fabric orientation
  5. Maps each image-space grasp point into calibrated table coordinates using a perspective homography computed from known table reference points
  6. Validates every Cartesian motion against the BracketBot URDF using damped-least-squares inverse kinematics
  7. Executes a full dual-arm sequence: right arm picks sock A, left arm picks sock B, both lift to safe height, converge to a central reunion pose, place the pair side by side, then retreat
  8. Escorts unmatched socks to the Singles Club
  9. Celebrates with a happy wiggle. The pipeline runs identically in simulation and on hardware via a clean backend interface.

Sole Mates also includes an optional Claude-powered recovery mode for simulated grasp-planning failures. When a grasp fails workspace or inverse-kinematics validation, Claude selects an alternative from supplied grasp candidates and provides a short rationale. The robotics planner validates the selection before continuing, with at most three recovery attempts. Recovery is disabled by default, preserving the original demo behavior.

How we built it

Tech Stack

  • Python 3.11
  • OpenCV 4.8: camera capture, background segmentation, connected components, HSV histograms, Laplacian texture, contour analysis, PCA orientation, distance transform grasp points NumPy: all geometry, feature vectors, IK math
  • Custom URDF Parser + Damped-Least-Squares Inverse Kinematics (solemates/kinematics.py): pure NumPy FK and numerical IK against the supplied chopped_urdf_v2 URDF, no external IK library required
  • BracketBot libhybrid_ik_lib.so: organizer-provided RelaxedIK solver (Rust, ARM aarch64) wrapped via ctypes for hardware deployment
  • Pinocchio: used in the BracketBot compliance controller for Jacobian computation and wrench estimation
  • SciPy: optional optimal pair assignment
  • Rerun: 2D/3D visualization, annotated frames, grasp points, and .rrd run recordings
  • Pytest: unit tests for kinematics, matching, and full pipeline
  • JSON config: all thresholds, workspace bounds, destinations, and backend selection in one file

Architecture

Overhead Camera (Webcam / DroidCam Phone) | v OpenCV Segmentation + Feature Extraction | v Weighted Pair Scoring + DP Global Matching | v Grasp Planner (Distance Transform + PCA) | v Pixel -> Table Coordinate Transform | v Damped-Least-Squares IK (7 DOF / Arm) | v +--------------------------------------------+ | | v v Simulated Robot BracketBot Hardware (Workspace + IK Validated)
| | +----------------------+---------------------+ | v JSON Event Log + Annotated PNG + Rerun .rrd

Simulated AI Recovery

Workspace / IK Failure | v Candidate Grasps + Failure History | v Claude Selection | v Existing Planner | v Workspace + IK Validation | +------> Resume Execution | +------> Retry (up to 3 attempts)

Challenges we ran into

  • IK from scratch: We implemented a full NumPy FK and damped-least-squares IK from scratch to parse joints, compute the Jacobian numerically, clamp to URDF limits, and handle the 7-DOF redundancy of each arm.
  • Real vs. synthetic perception.:The synthetic scene is perfectly lit on a uniform background. Real webcam images have shadows, variable lighting, and inconsistent backgrounds. Tuning background_bgr and background_distance for a real surface required iterative testing.
  • Camera connectivity: DroidCam on iOS connects via a Tailscale VPN address (100.x.x.x) rather than a local WiFi IP, which blocks direct OpenCV URL access. We pivoted to using the laptop webcam with a live preview tool for real-image testing

Accomplishments that we're proud of

  • 93%+ match confidence across all test runs, with correct pair identification and singleton detection every time
  • 33 IK-validated arm movements in a single full simulated run; every Cartesian pose checked against the real URDF before execution
  • Built a homography calibration pipeline that maps camera pixels to real-world table coordinates via a 3×3 perspective transform fitted to four clicked markers; the planner applies the homography to grasp positions and its local Jacobian derivative to sock orientations, so gripper yaw stays correct under lens perspective
  • Zero-training perception as the entire detection and matching pipeline uses classical computer vision with no neural network, no dataset, and no GPU
  • Optional zero-shot fallbacks built and tested: SAM (Segment Anything) for difficult backgrounds, CLIP embeddings for similar-pattern matching
  • Clean sim/hardware interface as SimulatedRobot and BracketBotRobot implement the same four-method interface, so swapping backends requires one config line change

Built With

Share this project:

Updates

Submission history