SortQuest

dean the dude
dean the dude
dean the dude
dean the dude
dean the dude
exit
background outside garage
trash guide board
player dropping item in bin
player grabs can
robot dropping item in correct bin
tutorial query interface
gripper selection interface
interface (with metrics)
robot arm collides and gets stuck showing red

Inspiration

Recycling only works when materials are sorted correctly, and a single lithium battery in the wrong stream can start a fire at a sorting facility. Robots can help, but they learn to grasp from examples, and good examples are slow and expensive to collect, especially for rare, dangerous items. People are naturally good at picking things up. So we built SortQuest: a VR recycling game where every successful grab becomes a demonstration a robot gripper can learn from.

What it does

Players stand at a conveyor belt in a virtual recycling facility and sort trash into four bins: metal (blue), plastic (yellow), paper (green), and hazardous (red). There are 24 item types (six per bin, 66 models in total), from cans and bottles to AA batteries and power banks. A trash guide board shows every item under its bin, and an optional narrated tutorial walks new players through their first sorts.

Every grab is recorded as a robot grasp relative to the item: where the gripper grabs, which way it closes, which way it comes in from, and how wide it opens. A grab only counts as a good demonstration if the item lands in the correct bin without being dropped.

Then the robot takes its turn. It uses the grasp that the most successful players agree on, checks it against the belt, the bins, and nearby trash before it moves, and sorts items with a simulated arm. Afterwards, it names the item it struggled with most and asks the player to teach it, then tries again, so new data goes where the robot is weakest. A live accuracy chart, confidence bars, and a ghost gripper show it learning, and the Results screen is read aloud.

Players choose among four grippers (standard, compact, and wide two-finger grippers, plus a suction cup), and each learns separately. Every robot attempt is saved with its outcome and the reason it failed, and an overhead camera saves color, depth, and item-mask images with each grasp.

How we built it

Engine and VR: Unity 6.3 (C#, URP) with OpenXR and the Meta XR SDK for Meta Quest, using hand tracking or controllers shown as hands. We tested with Quest Link and the Meta XR Simulator.
Hand to gripper: from the thumb tip pₜ and index tip pᵢ, the grasp center, closing direction, and width are

$$c = \frac{p_t + p_i}{2}, \qquad x = \frac{p_i - p_t}{\lVert p_i - p_t \rVert}, \qquad w = \lVert p_i - p_t \rVert$$

The approach direction runs from the wrist to the middle knuckle, with its closing-direction component removed. All of it is stored in the item's frame, so the grasp still applies wherever the item ends up.

Learning without averaging: the robot picks the medoid grasp,

$$g^* = \arg\max_i N_i$$

where Nᵢ is the number of good grasps within 2 cm and 20° of grasp i. Its confidence is the share of good grasps that agree with the chosen one.

A collision-aware robot: the whole gripper (fingers, housing, and wrist) is checked with physics queries along its path. Each finger pad must press on a surface facing it within 30°.
Practice: each good human grasp is tried with small variations on a hidden copy of the item, and only the variations that pass the robot's checks are kept.
Data: grasps are saved locally as JSON Lines, so the game never waits on the network. A FastAPI server uploads them to MongoDB Atlas with a retry queue. Human demonstrations are never altered: whether each gripper could perform one is stored as a separate note, and every label records which version of the checks produced it.
Voice: ElevenLabs narrates the tutorial and reads the Results aloud. Speech is generated through our server, so the API key never ships in the headset app.

Challenges we ran into

Hands aren't grippers. We turned a pinch into a center, a closing direction, an approach direction, and a width relative to the item, so it stays meaningful as the item moves.
Averaging grasps fails. The average of a grasp on the left of a bottle and one on the right is a grasp on empty air, so the robot picks a representative grasp instead of averaging.
Realism versus sloppy demonstrations. When we made the contact and collision checks stricter and more realistic, the robot started refusing almost everything, because many recorded pinches, especially from the simulator, were off-center. We kept the strict checks for data quality and made the robot's no-data guesses physically sensible instead.
Keeping data clean under interruptions. Players can quit mid-game. Unfinished grabs are discarded rather than mislabeled, and interrupted robot attempts are saved as "aborted" and never scored.
Hardware and teamwork. Quest Link didn't support one laptop's GPU, so much of our testing ran in the Meta XR Simulator. With four people changing one Unity scene, we relied on Unity's scene-aware merge tool.

Accomplishments that we're proud of

A complete loop in one playable scene: play, record, judge, learn, let the robot try, teach its weakest item, and watch it improve.
Honest, labeled data: successes and failures with reasons and stages, camera images, and check versions, while human demonstrations are never rewritten.
24 item types across four bins and four separately taught grippers, running on Meta Quest via Quest Link.

What we learned

A useful demonstration needs its outcome, not just a pose.
Strict, realistic labels are worth more than a flattering success rate. Grasps wrongly labeled as good teach a robot bad habits.
Tracked fingertips rarely land exactly on an object's surface, which is a known issue in hand-object research. So the robot verifies contact itself instead of trusting the pinch.

What's next

Collect clean demonstrations with real hand tracking on the headset, across all 24 item types.
Measure and optimize performance running on the Quest 2 alone, without a PC.
Snap recorded contact points onto item surfaces, and give more weight to accurate players.
Our simulation idealizes contact, with no slipping or forces. The data is best used to pre-train a robot, followed by fine-tuning with a small amount of real robot data, which will need real-world testing.