lock in.

User interface for lessons

FOR EVALUATORS/SPONSORS:

Scroll down for a handy guide to navigating our repository and our project's assets.

💥 How it all started

Every student learns differently. Traditional educational approaches, while effective for some, often fail to cater to the diverse needs and paces of individual learners. Even traditional AI learning is often one-sided lecturing that fails to properly engage the student in actual learning. This realization sparked a quest for more personalized and adaptive teaching strategies.

The capability of AI to analyze and interpret human facial expressions in real-time presented a unique opportunity to improve on today’s sparse learning environment. We envisioned a system that could monitor students' engagement levels, emotional responses, and cognitive states during lessons, allowing for dynamically changing teaching methods.

By leveraging Hume.AI’s expression detection, we aimed to create a feedback loop where we could have an AI educator receive immediate and actionable information about their students' learning experiences. This creates a seamless experience where the user does not even need to inform the AI about whether the teaching approach is right for them—they simply can focus on learning, while letting us do the rest.

The problems encountered by the rigid, traditional teaching methods:

Lack of Personalization: One-size-fits-all approach fails to address individual learning styles and paces, disproportionately affecting low-income and neurodivergent learners.
Student Disengagement: Inability to maintain student interest and motivation due to uniform delivery methods.
Limited Feedback: Current education practices struggle to gauge real-time understanding and emotional responses of all students.
Underutilized Potential: Advanced students may not be sufficiently challenged, leading to boredom and disengagement.
Teacher Workload: High demand on teachers to adapt lessons manually for diverse classrooms without sufficient support.
Resistance to Change: Institutional inertia and reluctance to adopt new teaching technologies and methodologies.
Rigid Schedules: Students must choose between real-time teaching with instant feedback (that they must attend at certain times) or asynchronous learning that does not guarantee instant feedback (as the instructor is not always online)

📖 What it does

To address these challenges, we developed lock in, an app designed to allow students of any age or level to learn the way that is best for them.

Here’s how lock in leverages advanced technologies to serve its mission:

Personalized Learning Assistants: By leveraging Hume’s Empathic Voice Interface, we are able to provide students the ability to learn from and converse with an interactive AI assistant that leverage RLHF (reinforcement learning through human feedback) techniques to adjust its teaching methods based on the students facial and vocal expressions.
Inclusive Assistant for Neurodivergent Learners: lock in was designed with neurodivergent learners in mind (if not at the forefront). Those with special needs in education (especially in developing parts of the world) often don’t have proper infrastructure for their learning. We especially aim to target those with ADHD, with our AI constantly watching and innovating on its education in order to keep those with shorter attention spans engaged in learning.
Intelligent Curriculum Generation: By integrating Mathpix and Claude 3.5 through Amazon Bedrock, lock in is able to generate a detailed curriculum to be taught by the AI learning assistant. In doing so, students can easily supplement their in-class learning with a curriculum that is based on their current learning.
Active Learning Encouragement: We designed our model to not simply lecture the student, but involve the student in the learning process by presenting them examples to grapple with and inviting them to connect concepts. Passive consumption only makes students feel like they’re learning, but results in less learning overall¹. We focus on inviting the student to actively grapple with material. lock in is more than just an education app; it's a movement towards personalized, efficient learning. Through the use of advanced technology such as React, FastAPI, AWS, Hume, and Claude, we aim to create a world in which learning is fun for everyone.

🔧 How we built it

To address the challenges of creating personalized education with our app, lock in, we've woven together a fabric of cutting-edge software technologies, each selected for its ability to enhance the app's functionality and user experience. Here's how each technology contributes to lock in:

Technologies:

React: Powers the user interface, offering a smooth and pleasing experience
FastAPI: Provides the backbone for the backend, handling requests efficiently and ensuring scalability to support a growing user base.
AWS Bedrock (Claude 3.5 Sonnet): Used to give more context about syllabi, especially when they contain images and diagrams that cannot be processed by Mathpix. This allows lock in to more easily understand what it’s looking at. Claude also creates comprehensive lesson plans and lesson topics, which are then handed to EVI for specifics.
Hume’s EVI (powered by Claude 3.5 Sonnet): Provides speech to speech voice chatting capabilities through EVI (Empathic Voice Interface), taking into account users’ voice tones and emotions. This allows EVI to learn how to better teach on the fly and change teaching styles if it detects boredom or other negative emotions. This switching of styles can be especially useful for neurodivergent students who may have trouble staying on target. EVI handles the specifics of teaching each lesson, handed to it by Claude on Bedrock.
Hume’s Expression Measurement API: Provides the ability for EVI to read facial expressions and emotions, giving even more emotional context. EVI will respond to speech in a more nuanced way, with even more information than it usually has by itself.
Mathpix: Extracting the text from syllabi, allowing it to be passed into Claude to generate a curriculum. This allows the user to easily and accurately import pre-existing lesson plans.

Learning with AI today is currently extremely hard and time consuming. But, with the integration of these emotionally aware technologies into lock in, we can create a more proactive and sensitive learning environment, suitable for those young and old, as well as those neurotypical and neurodivergent.

Our AI assistant takes more control of the curriculum, rather than taking a backseat and requiring the student to come up with prompts. React and FastAPI ensure a robust and efficient app infrastructure, while Mathpix and the multimodal Claude 3.5 on AWS Bedrock give lock in the ability to understand images and create curriculums Lastly, Hume’s EVI (powered by Claude) and Expression Measurement API do the emotional heavy lifting. This makes lock in not just a lifeless learning tool, but a supportive and initiative-taking academic companion. This synergy of technologies makes lock in a powerful platform for individuals of all learning styles and needs, taking in human feedback and approaching change in a more human way.

🚩 Challenges we ran into

Information Integration: lock in records and processes data from multiple sources, including, but not limited to, user speech, syllabi, and user-drawn visual resources. It was a huge challenge to manage our data, creatively integrate them into different services, and process them to quickly provide Samantha’s seamless, and human-like responses.

Prompt Engineering: Our control over model input and output is fairly limited, meaning that we have to get creative with things like config settings and context. Most of our ability to specify behavior lies in our system prompt, however. This creates a nearly bottomless creative task of prompting our teaching assistant to educate our users as best as possible. This is, of course, made even more complicated by the unique personalities and learning styles of different people. We strive to capture all learners, but there is no “one-size-fits-all” prompt.

🏆 Accomplishments that we're proud of

Fully implementing all the major features we planned to when we first started brainstorming Integrating both vocal conversations/vocal expression (EVI) and facial expression (Expression Measurement API) into one seamless product, which has not yet been fully explored

We improved on Learning with AI in the following ways:

Typical AI chatbot learning experiences revolves around students needing to take the initiative to split a potentially complex topic up into parts themselves and tracking progress on them. Our lesson-based learning helps students who prefer learning "on-rails", giving a more structured learning experience that is less daunting to those who are unfamiliar with the topics they are asking about. This makes it easier to stay on task, track progress, and understand how the current lesson fits into the topic as a whole.
AI chatbots revolve around inputs and outputs being types. This results in stilted, slow experiences that don't quite feel "real" to a user. lock in features speech input and output, giving a more quick, conversational experience. This is also a more natural way of communicating, allowing users to properly immerse themselves in learning as if they were talking to a tutor or teacher.
Most LLMs require a good bit of calibration and user input in order to receive the desired result. Most casual users are either not aware or not skilled at this, and thus they are not able to extract full potential from their learning. Samantha continuously tracks emotions through speech prosody and facial expressions, and will offer a change of gears to the student if she detects something isn't quite working.

📝 What we learned

Nicholas: I learned a lot about the strengths and limits of modern multi-modal LLMs like Claude 3.5. After experimenting with different prompts and tasks, I feel more confident on how I would use them in the future and I am excited to see how new models push the boundaries.

Leon: Participating in this AI hackathon showed me the unlimited possibilities of AI, and how it can improve so much in the world. Working on this project had a lot of various struggles, but ultimately, building together with my team was an amazing experience that allowed me to learn so much.

Ryan: This AI Hackathon was my first ever hackathon experience, so getting a hang of the process and conventions was a very novel experience. I got a far better sense of how applications are actually tied together.

Alex: I had tons of fun figuring out how to integrate all of these completely different technologies into one seamless application. It was a huge challenge to manage our data and figure out if and how things should connect to each other. This is the most data heavy project I have ever worked on, and I had a blast doing so!

✈️ What's next for lock in

Just like how a great trip has a great itinerary, we envision lock in’s future plans in phases.

Phase 1: Solidifying Our Roots

Phase 1 involves solidifying the use of the whiteboard, as well as having Samantha remember specific learning characteristics of students. We also want to improve our UI to create a more streamlined and user-friendly experience.

Phase 2: Branching Out

Phase 2 involves improving Samantha’s specific subject capabilities, especially her skills in mathematics. We may experiment with other image analysis strategies, using the multimodal Claude for understanding diagrams and Mathpix’s OCR capabilities for reading text. We also want to give her image generation capabilities to supplement explanations with visual examples and close the loop on image to image communication.

Phase 3: Take Off

Phase 3 involves partnering with established online education companies like Brilliant.org or Curiosity, improving their services and reaching a larger community of those who love to learn.

📋 Evaluator's Guide to lock in

Intended for judges, however the viewing public is welcome to take a look. Hey! We wanted to make this guide in order to help provide you further information on our implementations of certain programs and provide a more in-depth look to cater to both the viewing audience and evaluators like yourself.

SPONSOR SERVICES WE HAVE USED THIS HACKATHON

Hume
AWS Bedrock (Claude 3.5)
Mathpix

REFERENCE:

1: Deslauriers L, McCarty LS, Miller K, Callaghan K, Kestin G. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proc Natl Acad Sci U S A. 2019 Sep 24;116(39):19251-19257. doi: 10.1073/pnas.1821936116. Epub 2019 Sep 4. PMID: 31484770; PMCID: PMC6765278.