modifai

Inspiration

Despite it being the legal right of students to be included in the classroom, students with learning disabilities are often left out. To include students with learning disabilities, teachers have to create custom assignments based on student's Individualized Education Plan (IEP) that integrate with the flow of the larger class. However, teachers are low on time and resources to accommodate and include these students. In addition to lawsuits, this has led to student's with learning disabilities being left out and having exclusion---rather than inclusion---define their educational experience.

We designed an application to automate accommodating students, curating interactive learning experiences from existing assignments that adjust to students' needs according to their IEP with minimal input from the teacher.

See more about our team member's (Tyler) personal connection to our mission in the attached video.

What it does

modifai (pronounced modify) is a platform that automates transforming K-12 assignments to adjust to students with IEPs, turning plain assignments into guided interactive experiences that allow students to follow the class' curriculum without feeling left out while offloading burden off of the teacher.

We take two documents: (1) A student's Individualized Learning Plan (IEP); and (2) an assignment a teacher made, and generate a personalized assignment according to the student's IEP (personal learning plan) that fits their unique requirements for learning.

Given that having an assignment read aloud is one of the most common academic accommodations, students are also able to step through the modified assignment while hearing auto-generated narration along the way in their teacher's voice, with detection integrated to detect confusion and ensure attention and allowing the student to get questions answered by a fine-tuned LLM for the task when needed.

How we built it

React
Flask
Hume (for detecting when clarification is needed for a student)
OpenAI API
OpenAI Fine-tuning

Challenges we ran into

70+ page long IEPs to effectively parse to generate custom assignment modifications from
Constructing a good dataset for fine-tuning
Choosing the right parts of the document in order to provide feedback
Creating document specific techniques for creating semantically coherent chunks to provide appropriate context to the large-language model.
Merge conflicts
Committing too fast (OPEN AI API keys)
Lots of experimentation leading to a disorganized code base

Accomplishments that we're proud of

We designed a retrieval pipeline for documents, often longer than 75 pages, using an abstractive technique inspired by RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) -- effectively incorporating our passion for AI research into a real world use case that makes the world a more equitable place. We also designed a pipeline to segment a text into digestible logical units, so that we can generate annotations, vocal narrations --- this way the student can interactively walk through the assignment with instruction and helpful hints read aloud.

We believe the contribution of many apps that leverage language models (like ours) are in how well they reduce friction of use. We are incredibly proud of our user interface and believe it is a core component of the value of our system. Being able to provide so much functionality in a simple interface is a great achievement on top of the technical functionality we were able to create in 24 hours. Although we did not fully leverage synthetic data generation for fine-tuning, we developed a proof of concept for generating data to fine-tune GPT-3.5. This aimed to speed up our output generation while maintaining accuracy and producing more structured results.

What we learned

How to integrate Hume!
To check .gitignore for .env
How to construct a data set for fine-tuning

What's next for modifai.co

We have built proof-of-concept for our startup modifai in 24 hours. We are eager to get this product into the hands of teachers to make the classroom a more equitable place to learn -- including students across ability levels and improving educational outcomes. The modifai team seeks additional resources to get the world closer to our mission of making academic accommodation easy.

Who we are

Tyler Katz: Masters student at Carnegie Mellon University studying Computational Biology, interning as a Software Engineer at Genentech.

Julius Arolovitch: Incoming junior in Electrical and Computer Engineering & Robotics at Carnegie Mellon University with research interests in grasping and motion planning for manipulation. Summer intern at Johnson & Johnson working on Ottava.

Quentin Romero Lauro: Junior CS major at University of Pittsburgh, working in the EPIC Data Lab at UC Berkeley on techniques to improve retrieval-augmented generation pipelines.

Benjamin Kleyner: Sophomore CS major at Carnegie Mellon University, interning at Insitro as a software engineer.