workflow
website2
website1

Inspiration

The Performers Academy (TPA) works with video recordings of performances involving activities such as singing, dancing, acting, and spoken word. Reviewing long videos manually to identify specific activities, locate performers, and retrieve important moments can be time-consuming.

Our inspiration came from the opportunity to simplify this process by combining computer vision and machine learning techniques into a single platform. We wanted to develop a system that could automatically analyze performance recordings, organize activities, detect and track people, and generate meaningful reports that staff could easily review.

Our goal was to make video analysis more accessible, organized, and efficient for The Performers Academy.

What it does

TPA Insight is a multistage video analysis platform that helps The Performers Academy process, organize, and review performance recordings.

The platform provides the following capabilities:

Activity Recognition: Identifies performance activities such as singing, dancing, acting, and spoken word.
Video Segmentation: Divides recordings into timestamped segments based on predicted activities.
Person Detection: Detects people appearing in video frames.
Person Tracking: Associates person detections across frames within individual recordings.
Facial Expression Analysis: Provides model-generated facial expression suggestions.
Person and Activity Library: Organizes detected person tracks and activities, allowing staff to search and retrieve relevant segments.
Reporting and Review: Generates structured reports that staff can review, correct, and download.

Users can upload a video through the Streamlit dashboard, run the analysis, explore activity-based segments, view person-track details, and download selected video clips and reports.

How we built it

We developed TPA Insight using Python, integrating pretrained computer vision models and custom video-processing components into a multistage pipeline.

1. Activity Recognition — CLIP / X-CLIP

We integrated pretrained models available through the Hugging Face Transformers library to recognize performance activities from sampled video frames. CLIP serves as the default activity recognition model, while X-CLIP is supported as an alternative.

2. Video Chunking and Segmentation — OpenCV and Python

We used OpenCV to read and sample frames from videos. The pipeline processes recordings in overlapping 10-second windows with a 5-second stride. Custom Python logic then organizes the activity predictions into non-overlapping segments with start and end timestamps.

3. Person Detection — YOLO11n

We used the pretrained YOLO11n model through the Ultralytics Python library to detect people in video frames and generate bounding boxes.

4. Person Tracking — ByteTrack

We integrated ByteTrack to associate person detections across frames, helping us organize observations of individuals within each video.

5. Face Detection and Expression Analysis — SCRFD and FER

We used SCRFD for face detection and FER to generate facial expression suggestions from detected faces.

6. Dashboard and Reporting — Streamlit and SQLite

We developed an interactive dashboard using Streamlit and used SQLite for storing analysis results and staff reviews. The application supports video uploads, activity browsing, person-track exploration, report verification, and downloading selected clips.

Challenges we ran into

One of our biggest challenges was integrating several pretrained models into a single processing pipeline while maintaining consistent outputs across different stages.

Video processing efficiency: Analyzing long recordings can require significant computational resources. We used frame sampling and window-based processing to make the workflow more manageable.

Person tracking consistency: Tracking people across frames is challenging when performers move quickly, overlap, or temporarily disappear from view. Tracking identifiers can change, so we designed the application to treat them as video-specific references rather than verified identities.

Activity recognition accuracy: Performance recordings can contain multiple activities, and visual information alone may not always distinguish activities such as singing and speaking. We addressed this limitation by treating model predictions as suggestions that staff can review.

Combining analysis results: Organizing activity predictions, person detections, face observations, and timestamps into a consistent report required additional data-processing and integration work.

User interface design: Presenting complex video analysis results in a way that nontechnical staff could easily understand was another challenge. We refined the dashboard to make video navigation, activity browsing, and report review simpler.

Accomplishments that we're proud of

One of our main accomplishments is developing a working multistage framework that integrates several computer vision models into a single video analysis application.

We are particularly proud of:

Integrating multiple models: Combining activity recognition, person detection, tracking, face detection, and facial expression analysis into one workflow.
Activity-based video organization: Converting long video recordings into searchable, timestamped activity segments.
Person and Activity Library: Developing a feature that allows staff to explore detected person tracks, view associated video segments, and retrieve relevant clips.
Interactive dashboard: Building a Streamlit application that brings together video uploads, analysis results, search, and reporting.
Human-in-the-loop review: Allowing staff to verify and correct model-generated observations rather than treating every prediction as a confirmed result.
Video clip retrieval: Enabling users to preview and download selected video segments instead of manually searching through complete recordings.

We are especially proud that the project goes beyond individual model predictions and provides an integrated application designed around a real organization's needs.

What we learned

This project helped us gain practical experience in developing end-to-end computer vision applications.

We learned how to integrate pretrained models from different libraries, process video frames efficiently, manage timestamps, and organize multimodal analysis results.

We also gained a better understanding of the limitations of activity recognition, person tracking, and facial expression analysis, particularly when working with dynamic performance recordings.

Another important lesson was the value of user-centered design. A technically advanced system is only useful when users can understand its outputs and interact with them easily.

Most importantly, we learned that machine learning predictions should be accompanied by clear explanations, appropriate uncertainty, and opportunities for human verification.

What's next for A Multistage Framework for Performance Video Analysis

Our next goal is to further improve the performance, reliability, and usability of TPA Insight.

We plan to focus on:

Improving activity recognition: Evaluating additional pretrained video models and improving recognition of activities that look visually similar.
Enhancing person tracking: Reducing tracking inconsistencies when performers move, overlap, or temporarily leave the frame.
Optimizing video processing: Improving computational efficiency to support longer recordings.
Expanding search capabilities: Making it easier for staff to locate particular activities and relevant moments across multiple recordings.
Improving reporting: Developing clearer summaries, visualizations, and more efficient review workflows.
Real-world evaluation: Testing the platform with additional performance recordings and collecting feedback from The Performers Academy staff.

Our long-term vision is to develop TPA Insight into a reliable and practical video analysis platform that helps The Performers Academy transform performance recordings into organized, searchable, and meaningful insights.