Inspiration
We struggled using flashcards in college even though they were great memorization tools. We felt like we were unable to truly learn concepts and were encouraged to memorize instead of understand the terms. Especially for studying alone, we felt that developing a product that could provide more honest feedback on how we remembered terms would be a more effective way to learn.
What it does
The SmartFlash companion provides an interface to study a previously uploaded deck of flashcards on a given topic. This app will take in your responses to a flashcard prompt and tell you if you’re correct.
How we built it
We used semantic similarity models from Cohere to embed the definition and the response for a given question, and used cosine similarity to calculate this. However, we found that this wasn't always reliable when statements contradicted or were deliberate inversions of the definition. So, we incorporated an entailment contradiction classifier via Roberta and Hugging Face inference endpoint, which we factored in to an overall correctness score presented to the user.
In addition to Cohere and Hugging Face, we tried incorporating summarization/generation metrics like Rouge and BLEU (as we felt that these mirrored the task of generating text given a prompt), sentence transformers for embedding, and a few other things.
Finally, we used Streamlit to host our application and provide logic for the "flashcarding."
Thank you to Arjun for all of the programming and model integration!
Challenges we ran into
We really wanted to incorporate the AssemblyAI (AAI) real-time API, since we wanted users to speak their responses (which reflects interviews or oral exam practice) and have them immediately be graded in app. We ran into issues with errors from AAI and also our inexperience with using real-time packages.
Accomplishments that we're proud of
We were able to develop a working program based on metric weighting. To test the effectiveness of any given metric, we developed a list of questions and various responses (both correct and incorrect) to account for semantic differences, contradictions, and edge cases. We ran ~7 different answers against each question and evaluated the score produced by 5 different metrics and analyzed the data to determine the most consistent determinants of correctness. We then modified the weighting such that we could establish a standard threshold at which correct answers are deemed correct and incorrect answers were deemed incorrect.
Thank you to Ashton for compiling this data and analyzing it to calculate the correct weighting scheme!
What we learned
We learned about various models and metrics and how they're constructed/what they're looking for and how to develop apps using streamlit. We also learned that we could combine ML models that weren't meant for a given task but provided value, and build an ML application without needing to use any training data.
What's next for SmartFlash Companion
We think a logical next step is integration of the AssmeblyAI real-time transcription API to make this even more hands-off. This improvement could come with automated grading/auto-advance once an answer is determined correct.
Log in or sign up for Devpost to join the conversation.