Sendsei

Chess.com analysis but for climbing!

Comment

Inspiration

Climbers usually know where they fell, but not what they should change on the next attempt. The visible failure is often only the final result of a sequence that started breaking down several moves earlier.

Sendsei was inspired by game reviews like stockfish analyzing moves in chess: instead of showing only the final mistake, sendsei reconstructs an attempt move by move, identifies the earliest evidence-backed turning point, and turns that insight into one practical correction.

The fall is where the attempt ended. Sendsei finds the move that changed the send.

What it does

Sendsei is a opensource project phone-first bouldering review application. A climber records or uploads an attempt, confirms the holds in the route, and can optionally add a successful reference attempt for comparison.

Sendsei analyzes body movement, hold contacts, pauses, and transitions to divide the climb into meaningful phases. It then finds the earliest defensible key moment or divergence and presents it through a synchronized, game-review-style timeline.

The climber receives:

  • A move-by-move breakdown of the attempt
  • Pose and hold overlays on the original video
  • An evidence-linked turning point
  • A comparison with a successful attempt, when provided
  • One concise coaching cue
  • One practical drill for the next attempt

Every conclusion must reference real frames, phases, holds, or measurements. If there is not enough evidence, Sendsei qualifies its answer or abstains instead of inventing an explanation.

How we built it

The mobile experience is built with Next.js, React, TypeScript, and Tailwind CSS and deployed on Vercel. The browser handles camera capture, video validation, route-color selection, lightweight motion analysis, and adaptive pose-based framing feedback.

For computer vision, we use MediaPipe Pose Landmarker in the browser and a deterministic server pipeline built with Python, MediaPipe, OpenCV, and FFmpeg, running on Modal. This pipeline normalizes each video, extracts body landmarks, matches hands and feet to confirmed holds, segments movement phases, calculates conservative 2D movement metrics, and selects supporting evidence frames.

The resulting evidence is passed to the OpenAI Responses API with GPT-5.6. GPT produces a structured review using only the supplied measurements, IDs, and evidence frames. Its response is validated against a strict JSON schema, and the server rejects references to nonexistent holds, phases, frames, or metrics.

Videos and generated evidence are stored privately in Cloudflare R2, using direct and resumable multipart uploads. Supabase Auth and PostgreSQL provide authentication and durable application data.

We also created shared, versioned data contracts so the frontend, computer-vision pipeline, and AI review service could be developed independently. The AI layer is modular, making it possible for the open-source community to add more model providers without rebuilding the computer-vision pipeline.

We used OpenAI Codex throughout development to investigate architectural decisions, implement features, write tests, review code, and integrate the frontend, infrastructure, and CV systems.

Challenges we ran into

One of our biggest challenges was extracting trustworthy information from a single phone camera. Monocular video cannot reliably measure force, wall depth, grip quality, fatigue, or true 3D biomechanics. We therefore separated direct observations, calculated 2D proxies, and AI inferences. Sendsei deliberately avoids injury predictions, definitive causal claims, and uncalibrated “send probability” scores.

Comparing attempts was another challenge because climbers rarely perform moves at the same speed. Instead of aligning videos by timestamp, we align them using confirmed holds, contact transitions, and movement phases.

We also needed to prevent AI hallucinations. A convincing explanation is useless if it references evidence that does not exist. To address this, every generated review follows a strict structured-output contract and every evidence reference is validated before being displayed.

Finally, processing video from a phone introduced practical challenges involving orientation, large files, unreliable connections, backgrounded browsers, and limited mobile processing power. We combined lightweight browser analysis with private direct uploads, recoverable jobs, and intensive server-side processing.

Accomplishments that we're proud of

We are proud that we built a complete phone-to-review workflow without requiring a native application or desktop computer.

Sendsei supports both standalone attempt analysis and failed-versus-successful comparison. It combines real video, pose tracking, confirmed route holds, deterministic movement analysis, and GPT reasoning in one synchronized review experience.

We are especially proud of the evidence boundary we created between computer vision and generative AI. GPT can explain and connect the evidence, but it cannot silently invent new measurements or cite moments the pipeline never observed.

We also built resilient uploads, durable job recovery, private video access, versioned data contracts, and honest abstention behavior—all within a hackathon timeline.

What we learned

We learned that the most useful moment is often not the fall itself. By the time a climber loses contact, the important change may already have happened several moves earlier.

We also learned that asking users to confirm route holds is not a failure of computer vision. A quick confirmation step improves accuracy, builds trust, and makes the final review easier to understand.

Most importantly, we learned that generative AI is strongest here as a reasoning and communication layer built on deterministic evidence. Computer vision establishes what can be observed; GPT turns those observations into a clear explanation and a practical next step.

Building for real phone video also taught us that capture quality, privacy, upload reliability, and uncertainty handling are just as important as model accuracy.

What's next for Sendsei

Next, we want to improve the hold and contact correction tools, calibrate our movement thresholds using consented real-world climbing footage, and evaluate the turning-point ranking against attempts labeled by experienced climbers and coaches.

We also plan to improve performance across more physical devices, add direct deletion and retention controls, and explore personalized beta using factors such as height and reach.

Longer term, we want Sendsei to learn from a climber’s history and recognize recurring movement patterns while continuing to distinguish clearly between what was measured, what was inferred, and what remains unknown.

Built With

Share this project:

Updates