DEXTRA

Home page
Trainer
clinic

Inspiration

Medical students need steady, precise hands, but practice with real feedback is rare and usually needs expensive simulators. We wanted something that runs on the laptop you already have. We were also inspired by Steady from Hack the Valley, and asked what it would take to make the numbers trustworthy: a score you can check, and a tool that says so when it isn't sure.

What it does

A 3D robot copies your hand. You pick two fingers. The instrument follows the point between them, and the jaws open and close with the gap. Pinch, and it grips.
Any two fingers work. Pinch them together three times, or tap them on screen. The hand model always reports all five fingertips, even for a finger someone doesn't have, so the person picks, not the app.
Setup check. Hold still for 3 seconds. Java measures tracking, frame rate and shake, and decides pass or fail. Gemini looks at one camera frame and tells you what to fix, like a light behind you.
The drill. Hit five rings in order. You get four numbers, each with its formula: accuracy, smoothness, speed, and how much of the time you were tracked.
It refuses to fake a score. If you didn't finish, or your hand was tracked in under 80% of frames, the attempt isn't scored, and it tells you why.
Gemini coach. Gemini turns your numbers into two or three sentences of advice. It never decides a score.
Lost-hand help. If your hand leaves the camera, the arm greys out, the edge you left from glows and an arrow points the way back.
Clinic mode. A coach or teacher sees one person's progress over time: the trend in each number, a by-day view, and every refused attempt. It says on screen that it's practice evidence, not a diagnosis.

How we built it

Java + Spring Boot owns the webcam. OpenCV (through JavaCV) runs MediaPipe's palm and hand landmark models to find 21 hand points about 30 times a second.
Java works out the point between your two fingers and the gap, detects pinches, checks hits, and scores every attempt in plain, tested code.
A WebSocket streams each frame to the browser, where Three.js draws the robot arm. The arm solves its joint angles every frame so the instrument tip lands where your fingers point.
Gemini (gemini-2.5-flash, with 3.8 Flash as backup) does two jobs: explaining the setup check from one camera frame, and coaching from your numbers. With no key or no internet, both fall back to offline text from the same numbers.
Tiger Cloud stores every finished attempt in a Timescale hypertable, and clinic mode's by-day view uses time_bucket.
The site is React + Vite + Tailwind, with WebGL and animation components from React Bits.
57 automated tests cover hand tracking on real photos, scoring on paths with known answers, refusals, the pinch detector, saving history and the API.

Where Gemini is used

GeminiCoach.java: coaching after each drill (numbers only), and the setup check explanation (one camera frame plus the measured numbers). Gemini is told the pass or fail is already decided and not to change it.
SetupCheckService.java: grabs the frame and calls the coach.
What Gemini never decides: any score, any pass or fail, any refusal. Java does.

Where Tiger Data is used

AttemptLog.java: each finished attempt goes into the dextra_attempts hypertable (time column started_at), tagged with the person and the two fingers they used.
Clinic mode's by-day view runs time_bucket('1 day', started_at) with averages of only the scored attempts, so a refused attempt never turns into a number.
If Tiger Cloud can't be reached, DEXTRA saves to a local file instead, and the screen says where the history lives.

Challenges we ran into

In our first real test, the hand vanished for almost 13 seconds near the bottom of the camera. The palm leaves the picture before the fingertip does, and the tracker needs the palm. We moved the targets up, and added a live camera view, a glowing edge and an arrow so you can find your way back.
The newest Gemini Flash model was overloaded during the hackathon (errors, and 35 to 60 second replies). We measured it, switched to gemini-2.5-flash (about 1 second), kept the newer one as a backup, and made everything work offline too.
Two fingers for everyone: the model can't tell when a finger is missing, so detecting it was a dead end. Letting the person pick was the honest fix.

Accomplishments that we're proud of

The robot copying a pinch, live, from a plain webcam.
Tested with a real medical/nursing student from Trent University.
Every number comes with its formula, and the app refuses to score when it can't back the number up.
It works for hands that don't match the default.
57 tests passing.

What we learned

Trust comes from the numbers you refuse to show, as much as the ones you do.
Measure before you blame your code: our slow frame rate was dim light, and our slow AI replies were an overloaded model.
Design for the user who doesn't fit the default, before writing code.