VibeCheck

Inspiration 💡

In a digital world of distractions, it has become increasingly difficult for students to truly engage with teachers - especially with the advent of online learning. Contrary to popular belief, while online learning spiked in 2021, researchers suggest that online learning has yet to peak. As a lot of us found out during COVID, online learning poses unique challenges for students as barriers for distractions are reduced. Teachers and Professors have their own challenges as well, with many of our own Professors mentioning how much more difficult it is to gauge understanding and attentiveness when teaching online classes.

Although attempts by other online platforms have been made to try to solve this problem before, as students, we think many of them take the wrong approach! Instead of cultivating empathy and helping foster a relationship between a curious and fascinated student with an engaging teacher, lockdown and surveillance software makes our classrooms feel forceful and relationships hostile.

Seeing this unaddressed problem coupled with the recent advances in the capability of AI to recognize and help us understand human emotion ultimately inspired us to build VibeCheck which is a proof-of-concept for an engagement analytics platform we imagine teachers can use to learn more about how students respond to their teaching style with the goal improving teacher-student engagement. And unlike platforms before it, not only do we think students should have the choice to opt-in / out to the facial expression models our platform uses, we want there to be benefits to students too by having our platform provide them a copy of their own engagement graph to learn from for each class they consent to VibeCheck being used.

What it does 🤔

In this hackathon, we built a web-app proof-of-concept that demonstrates our vision for VibeCheck through a live demo that uses the Hume facial expression model to estimate a user's engagement with an educational video. This was intentionally built to model how a future version of VibeCheck could estimate a student's engagement through a video stream as they attend a virtual classroom.

The app creates a live engagement over time graph as the student watches a video. This was designed to model what students who opt into our platform would see and how it can be useful in providing analytics that can help them identify their learning habits and areas of study they may need to focus in.

In addition, the app also implements an average live engagement over time graph that's processed by combining the individual engagement graphs each student produces. The average graph is designed to give teachers a generalized response to their lesson, where they can learn about distinct points in which engagement drops to help them develop future lesson plans.

Tech-Stack

Our application also has an end-to-end encrypted chatting module powered by @ Company. The landlords can filter through all the requests and send requests to tenants. This chatting module powers our contract generator module, where the two parties can discuss a particular agreement clause and decide whether to include it or not in the final contract.

How we built it ️⚙️

Our modern and fast and elegant web application was built using Next.js.

We integrated libraries like react-vis to build a visually appealing and live updating graph along with react-webcam and react-player to help simulate our student's video stream and lesson/lecture in our demo.

We wanted our clients a sleek experience and have minimal overhead, so we exported network heavy and resource-intensive tasks like computing the aggregate engagement graph and running the ML model to our Intel Cloud Compute Instance, MongoDB, and the amazing Hume.ai Streaming API.

Design 🎨

We went for skeuomorphism inspired design for this project for a futuristic look that still felt grounded in reality. We utilized design tools like Figma and found inspirations from the Vercel design system to prototype our designs before doing any coding. This helped us get iterative feedback so that we spend less time re-writing code.

Challenges we ran into 🧱

Not only was the first time we integrated the hume.ai API into our project, this was our first exposure to using AI in any application in general. Although the API endpoint was easy to understand, we still had to explore a lot of resources and ask for assistance from representatives for advice on which models and metrics they recommended us look at for our goal of measuring engagement. After several iterations of testing different formulas for our "engagement" metric based on the capabilities of the hume.ai API's output of 52 expressions and learning that many of the expression metrics were non-deterministic, we found relying on the combination of "interest", "concentration", and "joy" expressions output from 0-1 to be the most accurate.

Accomplishments that we're proud of ✨

We were proud of finishing the project on time which was an extremely tough task since we started working on it quite late as our flight from San Diego (UCSD) to participate at CalHacks didn't get us here until Saturday afternoon. Moreover, this was the first Hackathon we'd ever attended in person for one of us, and the first Hackathon we'd ever attended all together for the other one of us. Despite this, we were able to add most of the features that we envisioned for the demo of this greater analytics platform we envisioned. Even more importantly, we learned a lot about new web libraries and how to integrate AI into our project to meet our needs! Last but not least, we bonded over everything that broke (especially our sleep schedules) and everything that didn't :)

What's next for VibeCheck 🚀

We want to expand this demo to a full analytics platform that can empower both students and teachers to teach and learn better! To do this, we are planning to transfer what we learned on analyzing a live video feed in a web app to integrating our engagement vs time graph generation directly in video chat apps like Zoom, Teams, and Google Meet. Furthermore we are planning to launch with university campus clubs since we can get user feedback on workshops that many students hold virtually. Last but not least we need a lot of API credits to make this happen!

Note — API credentials have been revoked. If you want to run the same on your local, use your own credentials.