Anact Ortho

Film Screen
HooperIQ Screen
Recruit Screen
Logo

Inspiration

We grew up loving basketball, but elite development tools are locked behind expensive staff and private budgets. High level programs have front offices, scouts, and dedicated analysts. High school gyms usually get a phone and a court.

The gap is massive: there are nearly 900000 U.S. high school basketball players, a shortage of roughly 50000 referees, and families spending upwards of 1000 dollars a year on youth sports while private training costs 50 to 150 dollars an hour. When vsHacks announced automation as a theme, the goal was obvious: don’t automate a spreadsheet, automate the court. We wanted a single phone to handle reffing, commentary, and scouting so players can get elite feedback without a full crew.

What it does

Anact Ortho converts a courtside phone or uploaded video into three automated roles:Referee: Calibrate four corners, and the system tracks out of bounds and scoring events with an automated whistle.Commentator: Generates play by play audio from game events using an offline first phrase engine and text to speech.Scout: Measures jump height, release velocity, creates court heatmaps, clips highlights, and builds shareable player cards.The system includes a live camera interface for the gym, a "Film Room" and "HooperIQ" module for game knowledge training, and a recruiting portal to process raw footage into scout profiles. Everything follows a tight loop: sense $\rightarrow$ decide $\rightarrow$ act, end to end.

How we built it

Anact Ortho converts a courtside phone or uploaded video into three automated roles:Referee: Calibrate four corners, and the system tracks out of bounds and scoring events with an automated whistle.Commentator: Generates play by play audio from game events using an offline first phrase engine and text to speech.Scout: Measures jump height, release velocity, creates court heatmaps, clips highlights, and builds shareable player cards.The system includes a live camera interface for the gym, a "Film Room" and "HooperIQ" module for game knowledge training, and a recruiting portal to process raw footage into scout profiles. Everything follows a tight loop: sense $\rightarrow$ decide $\rightarrow$ act, end to end.How we built itThe project is built as an npm workspaces monorepo:apps/web: Vite + React (landing page, Live view, Film Room, HooperIQ, Recruit portal)apps/mobile: Expo + VisionCamera with a custom native C++ frame pluginbackend/: FastAPI video processing workers, WebSocket live streams, and scouting APIspackages/vision + packages/core: Shared math utilities and game logicTo translate pixels to real world meters, the user taps the four corners of the court. The system solves an $8 \times 8$ Direct Linear Transform (DLT) for a planar homography $H$ that maps image points to court coordinates:$$\begin{bmatrix} x' \ y' \ w' \end{bmatrix} = H \begin{bmatrix} x \ y \ 1 \end{bmatrix}, \quad X = \frac{x'}{w'}, \quad Y = \frac{y'}{w'}$$Using these four correspondences, we construct the standard DLT system $A\mathbf{h}=\mathbf{b}$ and solve it using Gaussian elimination to avoid the overhead of NumPy on the hot path. Tracking the ball in meters rather than screen pixels is what allows for accurate out of bounds decisions.When the ball is occluded by hands or defenders, a kinematic predictor extrapolates its velocity using basic gravity models, damping the trajectory when tracking data shows it is near a player's wrists. A C++ SpatialEngine runs on the frame processor thread to handle point in quad calculations, scoring detection, and event cooldowns. These same rule sets are mirrored in TypeScript to ensure parity between the mobile edge and the web backend.Our primary architecture rule was keeping the camera thread isolated. Vision processing runs strictly in C++, while the React UI reads only the latest snapshot via a lossy frameBus. If the UI gets busy, it drops frames rather than stalling the 30 fps camera loop.

Challenges we ran into

Occlusion: Color tracking fails when a player holds the ball. We implemented kinematic recovery with wrist proximity damping to maintain tracking continuity during physical contact.

Real time on mobile: Running computer vision on the JavaScript thread caused severe UI lag. Moving the hot path to C++ worklets and designing a lossy frame bus kept the interface fluid.

Parity across runtimes: Mobile C++, web TypeScript, and the Python backend all needed to evaluate game rules identically. We resolved this by exporting shared logic packages to prevent engine drift.

Hackathon scope: Balancing film study, IQ training, and recruiting tools within a tight timeline was tough. We prioritized the live automation pipeline, structuring the other features as modular extensions of that core data.

Theme and product fit: “Automation” can sound corporate. We reframed it as replacing three human roles on a real court, not bolting AI onto a dashboard.

Accomplishments that we're proud of

We successfully built a complete computer vision system that runs smoothly on a mobile device at 30 frames per second without locking up the user interface. Translating raw screen coordinate pixels into dynamic physical metrics like jump height and release velocity allowed us to establish a reliable end to end pipeline that functions entirely on the edge.

What we learned

Automation isn’t “add an LLM.” It’s a closed loop: sense $\rightarrow$ decide $\rightarrow$ act, with clear failure modes.Homography turns computer vision into sports physics, meters beat pixels for fairness.Edge systems matter as much as models: thread isolation and backpressure decide whether the demo freezes.Offline first engines beat cloud dependent demos when Wi Wi dies mid pitch.Judges and teammates need the same story: theme fit, working demo, and one technical idea you can defend.

What's next for Anact Ortho

Next steps include integrating on device pose estimation via MoveNet or TFLite (and CoreML for iOS), layering an NPU accelerated ball detection model over the current color tracking system, and generating QR shareable scout cards linked to local match histories.

Every kid. Every court. Every jump, on the record. Thank you VSHacks for this opportunity!