GaitKeepr

What Inspired Us

Our project started from a real problem in our apartment.

My roommates and I recently got into running, and we all had the same goal: improve our running form in a way that supports long-term health, not just short-term performance. As we started running more consistently, we kept asking questions like:

Is my form actually good, or am I building bad habits?
Why does my shin / knee / Achilles hurt after certain runs?
What exercises should I do for that specific pain instead of guessing?

A lot of runners (especially beginners) either:

ignore discomfort until it gets worse, or
search online and get generic advice that is hard to trust or apply.

We wanted to build something that gives runners a better first step:

form feedback from video, and
targeted exercise guidance based on where it hurts.

That became GaitKeepr.

What We Built

GaitKeepr has two main features:

1) Video Form Scoring (Computer Vision)

Users upload a short side-view running clip, and the app analyzes running form to return:

an overall form score
interpretable metrics (cadence proxy, overstride tendency, torso lean, vertical bounce)
coaching tips for what to improve

We also generate a pose overlay video so users can see the body landmarks and limb connections used for analysis. This makes the system more transparent and less of a black box.

2) Pain-to-Exercises Coach (RAG Chatbot)

Users can describe pain or discomfort (for example, shin pain, knee pain, Achilles tightness), and the app returns:

general exercise suggestions
mobility / strengthening ideas
dosage and caution notes
citations from our curated exercise knowledge base

This helps runners get more specific guidance than a generic chatbot answer.

How We Built It

We built the project as a monorepo with a simple hackathon-friendly architecture.

Frontend

Next.js
React
TypeScript

The frontend provides:

a video upload panel for form scoring
a chat panel for the pain-to-exercises coach
score / metric / tips visualization
polling for async video analysis results
source / citation display for RAG responses

Backend

FastAPI (Python)
Uvicorn
Pydantic schemas
local worker subprocess pattern for video jobs

The backend handles:

POST /upload (video upload)
GET /results/{job_id} (polling / results)
POST /chat (RAG responses)

For video processing, we use a job-based flow:

upload video
create a job_id
spawn a worker process
process video in the background
poll results from the frontend

This kept the UI responsive and made the demo more reliable.

CV / Form Scoring Stack

OpenCV (cv2) for reading/writing videos and frame processing
MediaPipe Pose for pose landmark detection
FFmpeg for browser-safe MP4 re-encoding (so overlay videos play in the UI)
custom heuristic scoring rubric for interpretable form scoring

The CV pipeline:

detects landmarks frame-by-frame
computes running-related features
converts them into scores with a transparent rubric
generates coaching tips and an overlay video

RAG / Pain Coach Stack

Actian VectorAI DB (running in Docker) for vector retrieval
Sentence Transformers (all-MiniLM-L6-v2) for embeddings
curated JSON exercise knowledge base (running-specific)
custom retrieval + response assembly logic in FastAPI

The RAG pipeline:

parses user pain description
extracts simple signals (body area, side, intent)
embeds the query
retrieves relevant exercise entries from Actian Vector
returns grounded guidance + citations

Infrastructure / Runtime

Docker + Docker Compose for local vector DB setup
local file storage for uploaded and overlay videos
local job store (jobs.json) for hackathon-speed reliability
Databricks SQL integration (teammate’s work) for backend metadata/results persistence and smoke-tested table operations

What We Learned

This project taught us a lot about both engineering and product design.

1) Reliability beats complexity in a hackathon

A simpler, interpretable system that works consistently is more valuable than an ambitious system that fails during demo. We prioritized:

stable APIs
clear fallbacks
debuggable outputs
visible overlays for trust

2) Interpretability matters for user trust

Users trust feedback more when they can see what the model is doing. Showing the pose overlay made a huge difference because users could connect:

visible joint tracking
measured metrics
final score and tips

3) Environment management is a real engineering challenge

Combining:

CV libraries (OpenCV / MediaPipe)
backend APIs
RAG stack / vector DB tooling

created dependency conflicts (especially around Python package versions). We learned the importance of separating concerns and isolating environments for different parts of the system.

4) Good AI UX includes failure modes

Not every video is usable. Instead of pretending every upload can be scored, we added clear fallback behavior for cases like:

no person detected
poor lighting
non-side-view angle
body not fully visible

This made the app feel more honest and usable.

5) RAG works best when the knowledge base is curated

The quality of the chatbot depended heavily on the quality and structure of the exercise entries. A smaller, focused, runner-specific knowledge base produced better and more relevant guidance than a broad generic dataset.

Challenges We Faced

1) CV + RAG dependency conflicts

One of the biggest issues was getting the CV stack and the chat/RAG stack to coexist cleanly in development. Some package combinations caused runtime import problems. We worked around this by using a more robust local setup and keeping the CV worker path isolated.

2) Browser video playback compatibility

Even when the overlay video was generated successfully, browsers sometimes would not play it because of codec/container compatibility issues. We solved this by adding an FFmpeg re-encode step to generate browser-safe MP4 output.

3) Async processing and job state consistency

Video processing takes time, so we needed a polling-based workflow. During development, we hit issues where jobs were created but results were not written correctly due to worker/runtime failures, causing 404 or stale states. We fixed this by improving:

worker spawning
local job status updates
fallback logic
frontend polling behavior

4) Balancing smartness with demo speed

There are many ways to make running analysis more advanced, but we had limited hackathon time. We chose a practical pipeline:

pose estimation
feature engineering
heuristic scoring

instead of trying to train a complex end-to-end model.

5) Frontend / backend merge conflicts during rapid development

Because multiple team members were editing related UI/backend files (especially around the upload and chat panels), we ran into merge conflicts and integration issues. We learned to:

keep API contracts stable
test end-to-end after merges
isolate feature branches more carefully

Next Steps

We see this project as a strong prototype, and there are a lot of exciting directions to take it next.

1) Improve CV scoring accuracy

calibrate heuristics using more labeled running videos
better normalize metrics for body size / camera distance
detect more gait features (foot strike, hip drop, symmetry, arm swing)

2) Real pose-overlay scoring confidence and quality checks

add confidence indicators (e.g., landmark visibility / tracking quality)
reject low-quality videos earlier (before full processing)
provide specific upload guidance (camera angle, lighting, framing)

3) Personalized feedback over time

track a runner’s scores across sessions
show trends (posture, cadence, stability)
suggest targeted improvement plans over weeks

4) Expand the exercise knowledge base

add more pain scenarios and progressions
include beginner vs intermediate versions
improve exercise dosage and return-to-run progression guidance
support more context (terrain, mileage changes, training load)

5) Stronger safety and triage behavior

better detection of red-flag symptoms
clearer escalation messaging (when to stop running / seek medical care)
improved disclaimers while keeping guidance useful

6) Better production architecture

replace local file storage/jobs with cloud storage + queue workers
move from hackathon polling to more robust background task processing
add observability/logging for worker failures and latency

7) Mobile-first UX

Since runners often record and review videos from their phones, a mobile-first upload and feedback flow would make the product much more practical in the real world.