NoteSync AI: Effective Learning with AI-Powered Notes

Inspiration

Imagine a student with ADHD or a visual impairment trying to keep up with a fast-paced lecture. Taking notes, focusing on the professor’s explanations, and keeping up with class materials can become overwhelming, creating barriers to learning. Our inspiration comes from wanting to create an accessible, AI-powered solution that reduces these challenges for students with disabilities—while benefiting all students by making note-taking easier and more efficient.

What it does

Our platform provides AI-powered note-taking that integrates multiple inputs—audio, visual, and handwritten—into organized digital notes. It transcribes lecture recordings, recognizes handwritten notes, interprets mathematical formulas, and generates quizzes and flashcards, all while syncing content in real-time.

How we built it

Frontend: Developed using the Streamlit framework for an intuitive, user-friendly interface. Backend: Implemented with RAG (Retrieval-Augmented Generation) for AI question answering, leveraging GPT for contextual note-taking. DSPy Prompt Optimization: Optimizes prompts for real-time responses and accurate data processing. AWS Services: Utilized for storage, processing, and real-time data synchronization. CoT Prompting: Chain-of-Thought prompting to enhance problem-solving and structured reasoning.

Challenges we ran into

Real-time Data Synchronization: Integrating audio, visual, and text inputs in real-time posed a significant challenge. Ensuring that handwritten notes, lecture recordings, and blackboard images were aligned accurately without lag was critical for maintaining the flow of information.

Balancing Frontend Usability with Backend Complexity: While the backend handled complex tasks like AI-driven transcription, formula recognition, and context generation, the frontend needed to remain simple and user-friendly. Bridging the gap between a powerful AI system and a smooth user interface was a delicate balance.

Accessibility for All Students: Making the platform fully accessible to students with ADHD, visual, hearing, or mental disabilities required designing custom features like real-time audio transcription, adjustable text formatting, and AI-generated flashcards while keeping it intuitive for all users.

AI Accuracy and Formula Recognition: Converting complex mathematical formulas and visual content into readable text and structured notes was difficult, especially when combined with real-time context understanding. Ensuring the accuracy of this feature across diverse subjects was challenging.

Optimizing Performance for Real-Time Use: Ensuring that all AI processes, including RAG (Retrieval-Augmented Generation) for answering questions and audio-to-text alignment, were optimized for real-time interaction without overwhelming system resources was a key technical hurdle.

Accomplishments that we're proud of

We successfully integrated multiple data formats into a cohesive platform, making note-taking more inclusive and less burdensome. Created AI-generated questions that students can use for personalized studying. Developed a time-stamped synchronization system that aligns lecture audio, visuals, and notes, improving the learning experience for everyone.

What we learned

We learned the importance of balancing user-friendliness with complex AI-driven features to ensure the platform remains intuitive while providing powerful functionality. On the frontend, we designed an interface that organizes various inputs—like audio, images, and notes—seamlessly, allowing users to interact with AI-driven insights without being overwhelmed. Meanwhile, the backend optimizes real-time data processing, including lecture transcriptions, image recognition, and formula conversion, ensuring smooth and immediate responses.

Connecting the frontend and backend efficiently was key. We needed to ensure that users experience real-time updates as they upload audio or images, with the backend AI processing these inputs instantly and presenting them in a clear and structured format. This balance between real-time AI performance and a clean user interface was crucial to making the platform truly effective and accessible for students, particularly those with disabilities.

What's next for NoteSync AI

We plan to introduce:

Personalized Study Plans: AI-generated based on students' notes, syllabus, and learning pace. Interactive Syllabus: Linking each syllabus topic to corresponding sections of notes, additional readings, or video content. Advanced Accessibility Features: Further enhancements for students with hearing, visual, and mental disabilities. Expanded AI Capabilities: Context-aware assistance and improved formula recognition to support more complex subjects.