NFTeach: Engaging Video Conferencing

• 

  Because lectures should be tailored and unique

Inspiration

Good professors are suffering from remote learning. How? They don't know how students are reacting to lectures. Social media influencers have all the tools in the world to boost their own engagement, so our goal was to give professors the same tools in remote learning.

What it does

It's video conferencing, similar to Zoom. The key difference in our approach is that we use facial recognition technology to detect facial expressions in students, and professors are notified (anonymously) when such detections arise. Our hack has two important tools for the professor:

1. Live feedback: similar to live stories, professors can see in real time the engagement levels and reactions from the class. A sudden high influx of "happy" faces will notify the professor. This gives the professor to "read the room" once more, and they can tailor their instruction accordingly
2. Post-lecture feedback: we also keep a running count of total happy expressions per lecture. This is similar to regular posts on social media. For each lecture, they can see how many total "likes" they received. With this information, they can compare different lectures to see what works and what doesn't. What did they do differently in one lecture that garnered a lot of engagement, and what can he/she do to keep that up? These are the types of questions that our hack aims to answer for the professor.

How we built it

We are using the Google Cloud Vision API to detect facial expressions, WebRTC to allow for media streaming and capture, and Vite for local hosting.

The first thing we had to do was figure out how to enable media streaming and capture via webcam. We opted to use WebRTC for its simplicity. With this, we were able to use the webcam and take captures every few seconds.

Second, with every capture, we use Google Cloud Vision API to detect facial expressions and log the data. Referencing the above important tools, this data is used in two ways. One is to notify the professor when a sudden influx of expressions come in. The other way is by keeping a running total count that the professor can view and compare with other lectures later.

Third, we used Vite as a local hosting app and integrated our components into it.

Lastly, we spent time prettying-up the hack to make it as UI/UX friendly and aesthetic as possible.

Challenges we ran into

Using Google Cloud Vision API was the biggest challenge; we nearly gave up on the project when we couldn't properly utilize it. We couldn't figure out how to send POST requests to the server. Eventually, however, we were able to figure it out and gain key access and authentication.

Accomplishments that we're proud of

We are proud of being able to use WebRTC and Google Cloud services, because we never worked with such technology before. It was rough at first, but taking the time to learn a new technology and successfully implementing it in such a short time frame was highly motivating for us. Of course, we're also proud of the project itself. With a hack like this, we believe we can give professors the tools they need to better instruct their classes.

What we learned

Web development as a whole is something our entire team isn't quite accustomed to. Having to learn javascript while also trying to incorporate more advanced technologies like Google Cloud Services was challenging, but by the end of it, we came out on top.

We also learned to never give up. In fact, this hack was our second attempt at project after having to abandon our first one roughly 8 hours in. After the switch, we were still standing in front of a giant wall in terms of the technologies we wanted to utilize. Slowly but surely, however, we were able to chip away through the wall and make it through to the other side.

What's next for NF-Teach: Engaging Video Conferencing

We would work on implementing multiple clients and cameras. In order to do this, we would have to upload our hack to a hosting platform. For the purposes of our demonstration, we mimicked the functionality by jamming all of our beautiful faces into one camera. This way, we could still log data of facial expressions for each person.

Another thing that's in store for the future is more advanced detection features. As of now, the key expression that we leverage is "happy". We would like to get to the point where we utilize machine learning to detect more subtle expressions and give the professor more detailed data. For example, this technology could read the expressions of each student and output to the professor--for example--that 34% of the class is probably showing inattention.

Built With

• google-cloud-vision-api
• javascript/html/css
• vite
• webrtc