TEAiCH - Youtube-like analytics for presentations

TEAiCH helps you improve your presentations by providing actionable feedback through analysis of your speech, slides, and audience engagement, using Pixtral. It has a dashboard and a chatbot.

Comment

Inspiration

Have you ever been in a lecture where half the class is doodling, and the other half is sneaking looking at their phones? We've all been there. As students and educators, we realized that teachers often have no idea when they're losing their audience. We thought, "What if we could give teachers superhero-like insights into their students' engagement levels?" That's when TEAiCH was born—a tool to help presenters not just talk at their audience, but truly connect with them.

What It Does

TEAiCH is like having a personal presentation coach that works in real time. It analyzes three key aspects:

  • Audience Attention: Using a simple webcam, it gauges the audience's attention and emotional responses.
  • Speech Analysis: It transcribes the presenter's speech, picking up on keywords and sentiment.
  • Slide Insights: Every time you switch a slide, TEAiCH captures it and extracts key information.

By combining these data streams, TEAiCH identifies moments when attention dips and tells you exactly what was on the slide and what you were saying. It then provides actionable feedback to help you tweak your content or delivery for maximum impact.

How We Built It

We divided this project into several sections:

  • Real-Time Emotion Detection: Leveraging OpenCV and deep learning models to process video feeds and interpret facial expressions.
  • Speech Transcription and Analysis: Using speech-to-text and mistral models to extract sentiments and keywords.
  • Slide Capture and Processing: Implementing keypress listeners to detect slide changes and using Pixtral to analyze slides.
  • Data Integration: Syncing all the data using timestamps to create a cohesive story of the presentation.
  • Feedback Generation: Developing a chatbot that connects to the data and gives you valuable insights on the data.
  • Dashboard Creation: Building an interactive dashboard with Streamlit to visualize attention levels, slide effectiveness, and more.

Challenges We Ran Into

Let's just say, merging video, audio, and slide data in real time is no walk in the park.

  • Data Synchronization: Aligning different data streams accurately was like herding cats, but with timestamps.
  • Processing Power: Real-time analysis is heavy on resources. We had to optimize our code to keep everything running smoothly on a local machine.
  • Emotion Detection Accuracy: Dealing with varying lighting conditions and camera qualities made facial recognition tricky.
  • Time Constraints: It's a hackathon—time flies when you're debugging!

Accomplishments That We're Proud Of

  • End-to-End Functionality: We built a working prototype that integrates multiple complex systems within the hackathon timeframe.
  • Real-Time Analytics: Achieving real-time processing of emotions, speech, and slides.
  • User-Friendly Interface: Creating an intuitive dashboard that presents complex data in an accessible way.
  • Live Demo: Successfully using TEAiCH during our pitch to showcase its capabilities live. Talk about using our own product!

What We Learned

  • Team Synergy is Key: Combining our diverse skill sets and dividing it into parallelizable tasks was crucial.
  • K.I.S.S (Seriously): We learned that we should keep to the core features and avoid shiny cool features that distract us from the mission.
  • The Power of Integration: Individually, data streams are just noise. Integrated, they tell a compelling story.
  • Time Management: Hackathons are sprints, not marathons. Prioritizing features and quick decision-making saved us.

What's Next for TEAiCH

We're dreaming big! Imagine TEAiCH becoming the YouTube Analytics for presentations:

  • Enhanced Analytics: Incorporate more nuanced emotional states and accurate attention data for deeper insights.
  • AI-Powered Coaching: Use advanced AI agents to provide more personalized and accurate tips.
  • Scalability: Optimize the system for larger audiences and online presentations.
  • Local Running: Run the app 100% local to keep privacy

In a nutshell, we believe TEAiCH has the potential to revolutionize the way presentations are delivered and received. And we're just getting started!

Built With

  • langchain
  • mistral
  • opencv
  • pixtral
  • streamlit
Share this project:

Updates