Class.ify

Inspiration

K-5 students face many obstacles when it comes to their learning in school environments, both inside and outside the classroom. Many young students are often unable to communicate their emotions or points of confusion to their parents and teachers, something that is especially true for students with developmental disabilities like Asperger's. In fact, recent studies have showed that 90% of parents want to know more about how their student is doing in school, but less than 10% believe they have the resources to do so.

Aside from difficulties in learning, young students also have to face significant challenges related to safety within school environments. Instances of bullying, which can be especially distressing for children who struggle with communication, often go unnoticed or unreported. Moreover, the sad and alarming rise of active shooter threats in elementary schools across the United States has created increasingly stressful environments for both students and their parents alike.

Class.ify is a smart, AI-powered application that addresses both these issues cohesively using a variety of tools and utilities (explained further in next section).

What it does

Parents want to know what's going on in school

Class.ify uses a variety of tools to help communicate students' learning inside the classroom. We give the teacher the option to record their and their students' voices (with consent) through our interface. After all the recordings for one day are submitted, Class.ify will summarize classroom content across four subjects: math, science, history, and english. Furthermore, we use a transcribed version of the recorded audio to note student progress across different learning goals, which is determined from classroom engagement and voice intonation when the student is speaking.

We help create a safer environment where everyone's on the same page

To limit privacy breach, Class.ify uses already-existing camera + audio systems in school hallways to detect unsafe environments. Using Hume VocalBurst description + emotion classification, the interface is able to flag potentially unsafe environments so that administrators and parents are able to respond to the situation as soon as possible.

Then, a summary of the flagged media feed is generated, which can be used to alert local authority (depending on the severity of the situation) and/or school staff.

How we built it

We used a variety of the tools and utilities introduced us at this hackathon to build our project.

To summarize inputted audio recordings, we use a pipeline consisting of OpenAI's speech-to-text API along with AWS Bedrock (an LLM) to succinctly summarize the most important parts of inputted media. To flag unsafe school environments and help parents/teachers identify where their student might need more help, we make use of Hume's great sentiment analysis API. More specifically, we use their VocalBurst model to capture expressions and emotions in input audio, which we then post-process and fine-tune for our use case.

Our application structure itself consisted of a Flask backend paired with a next.js frontend. We created a REST API in flask that allows the interface to query our AI endpoints as well as retrieve sentiment analysis to display to the user.

Challenges we ran into

Aside from the time constraint of having to build such complex application in 24 hours, we found it challenging to try to optimize how we use the APIs provided to us for our use case. For example, we tried several of Hume's models (like Face, Progidy, and VocalBurst) in an attempt to determine which worked the best in the context we provided. We found that the Face model was not the most well-suited to this situation, since hallway footage is often blurry, and is unable to capture details in students' expressions. Given this obstacle, we decided to use a VocalBurst model (along with additional post-processing) to try to capture expressions and sentiments expressed verbally.

Another large challenge we grappled with was the concern of privacy, as we didn't want parents, kids, or teachers, feeling uncomfortable with being recorded in the classroom. However, looking at recent studies, > 70% of parents approve use of camera in the classroom for academic purposes, such as lecture capture. Furthermore, this would be solely a listening device through a familiar item such as Alexa and geared towards children who are much younger (K-5) where parents are more involved in their education. Finally, the tool is not meant to capture individual student performance and penalize them negatively in any way; it's meant to serve as a way to facilitate parent to teacher communication, something that would undoubtedly prove useful for both sides.

Accomplishments that we're proud of

We had a bunch of features we wanted to implement, so I'm proud that we were able to narrow that list down and focus on developing a few really in-depth instead of focusing on too many. I'm also proud that we managed to use a variety of LLM's and APIs in a way which all where extremely useful to the project rather then just squeezing them in for the sake of it.

What we learned

I think the biggest challenge that we had wasn't technically, but debating that privacy aspect and how we were going to build our platform around it, so that was a really cool challenge and taught us a lot about how members of the industry have to deal with these ethical issues on a daily basis.