Project Name: LearnItLive

Inspiration

Our inspiration for this project came from the desire to bridge the gap between physical interactions and virtual learning. We wanted to create an intuitive tool that allows users to interact with their surroundings, gain feedback, and also expand their knowledge in areas like math using an automated voice-over and visual explanations. We were motivated by the potential to provide seamless, interactive learning experiences for both physical objects (like Raspberry Pi projects) and academic topics like math.

What it does

This project has two primary features:

1. Physical Object Recognition & Setup Guidance: The program can analyze and identify objects in the user's environment using GPT Vision. It provides feedback and suggestions on how to use or set up various items, like a Raspberry Pi, offering detailed instructions based on the detected object.
2. Math Learning via Voice and Visual Explanation: The program allows users to learn specific math topics by generating a voiceover and creating an MP4 video with animations using Manim. This provides a visual and auditory learning experience that explains complex mathematical concepts step-by-step.

How we built it

We used the following technologies and tools to build our project:

• GPT Vision: For analyzing the user's physical space and identifying objects. This allows the program to provide accurate suggestions and guidance on what users can do with their surroundings.
• Manim: For creating animated explanations of math topics, turning them into MP4 files with clear visuals and voiceovers.
• Python: To combine both functionalities, integrate GPT Vision and Manim, and handle the user interactions seamlessly.
• Speech Synthesis Tools: For generating the voiceovers based on the math explanations.
• FFmpeg: For compiling the final MP4 videos with voice and animation.

Challenges we ran into

1. Object Recognition Accuracy: The accuracy of GPT Vision in recognizing various physical objects in different lighting conditions and environments was a challenge. It took some time to refine the prompts and the context to get reliable feedback.
2. Voice Synchronization: Synchronizing the voiceover with the Manim-generated animations took a bit of trial and error. Ensuring the voiceover matched the pace of the visual explanation required fine-tuning.
3. Rendering Performance: Manim's rendering process for math animations can be slow, and combining it with real-time user queries posed some performance issues. We had to optimize the workflow to ensure the app was responsive.

Accomplishments that we're proud of

• Integration of Two Complex Features: Successfully combining object recognition and automated math learning into one cohesive program was a major achievement.
• User-Friendly Experience: The ability for users to interact naturally with the program (whether asking about an object or learning math) felt intuitive and engaging.
• Customization: We were able to allow the program to generate personalized learning materials, tailored to each user’s request, making it a versatile learning tool.

What we learned

• Interdisciplinary Integration: We learned a lot about integrating AI-driven object recognition with mathematical visualization tools. The challenges involved in combining these two areas opened our eyes to the power of cross-disciplinary innovation.
• User Interaction: We learned that creating an easy-to-use interface that users can intuitively navigate is essential, especially when combining complex tasks like real-time object recognition and math education.
• Performance Optimization: We gained experience in optimizing performance, especially in rendering and generating content in real-time, which is crucial for maintaining a smooth user experience.

What's next for the project

In the future, we aim to:

• Expand Object Recognition: Improve the accuracy and range of objects that the program can identify, potentially adding support for more specific categories (e.g., different types of electronics, tools, etc.).
• Enhance Math Content: Add more mathematical topics and advanced features like quizzes, step-by-step problem solving, and user feedback to improve the educational experience.
• Mobile Version: Build a mobile version of the app to make it even more accessible for users on-the-go.
• Collaborative Learning: Add features for group learning or study sessions, where multiple users can ask questions and share knowledge.

Built With

• computervision
• gpt
• manim
• python
• yolo