Mind Tutor

Authentication
Landing Page Stack Section2
Landing Page Stack Section1
Landing Page Stack Section3
quick access for guest
chatbox temporal session for guest (30min)
get cam & mic access
get share screen access
companion AI will automatically ask you when you confused which predicted by vision model
thinking too long will be passed to confused
When things get blurry, AI tutor clear the fog together. Think of tutor as your co-learner through voice-to-voice, just like two friends

Inspiration

Recently, I’ve found myself getting distracted by many external factors, which has significantly reduced my productivity. This personal struggle inspired me to think about how technology could help people stay focused and regain control over their attention while working or studying.

What it does

My system acts as a smart assistant that monitors user behavior on their PC. When it detects signs of distraction, it gently reminds the user to refocus. If the system senses that the user is confused about what they are reading, it proactively offers help. The agent tracks the content displayed on the screen and combines it with external documents uploaded by the user. Using this combined context, it can answer questions, clarify concepts, and provide relevant explanations in real time.

How we built it

We built Mind Tutor by integrating multiple components, including screen content tracking, natural language processing, and a contextual AI assistant. The system continuously analyzes on-screen text and user interactions to infer focus levels. We also implemented a document ingestion pipeline that allows users to upload external materials. These documents are processed and indexed so the AI can retrieve relevant information when answering questions. The overall system combines real-time monitoring with retrieval-augmented generation to provide helpful and context-aware responses.

Challenges we ran into

One of the biggest challenges was accurately detecting when a user is distracted versus when they are simply pausing or thinking. Another difficulty was ensuring that the system provides helpful suggestions without becoming intrusive or annoying. Additionally, integrating real-time screen tracking with external document understanding required careful optimization to maintain performance and responsiveness.

Accomplishments that we're proud of

We are proud of creating a system that feels proactive yet supportive rather than disruptive. The ability to combine on-screen context with user-provided documents to deliver meaningful assistance is a key achievement. We also successfully built a foundation for a personalized learning assistant that adapts to user behavior in real time.

What we learned

Through this project, we learned how challenging it is to model human attention and cognitive states. We also gained experience in building systems that balance automation with user comfort. Technically, we deepened our understanding of real-time data processing, context-aware AI systems, and integrating multiple data sources into a unified assistant.

What's next for Mind Tutor

Next, we plan to improve the accuracy of distraction detection and make the assistant more personalized over time. We also want to expand support for different types of content, such as videos and interactive materials. In the future, Mind Tutor could evolve into a full productivity companion that not only helps users stay focused but also optimizes how they learn and work every day.

Built With

chatgpt
chroma
docker
elevenlabs
fastapi
git
github
opencv
python
torch

Submitted to

LotusHacks x HackHarvard x GenAI Fund Vietnam Hackathon

Created by

My name is Phạm Nguyễn Gia Huy, and I am currently a second year student majoring in Computer Science at the University of Information Technology (UIT). During this 36 hour hackathon, I took on the role of Frontend Developer for our project, Mind Tutor, and to be honest, these 36 hours felt like an entire journey compressed into a single moment in time. It was challenging, sometimes overwhelming, but also incredibly meaningful. There were moments when everything felt uncertain, when things did not work as expected, but there were also moments when ideas came together beautifully, and those moments made everything worth it.
In my previous competitions, I usually worked as an AI Engineer, focusing on building intelligent systems behind the scenes. However, this time I made a conscious decision to step into a completely different role and explore UI and UX development. I wanted to understand not just how intelligence functions internally, but how it is experienced, how users interact with it, and how it creates a sense of connection. This shift pushed me far outside of my comfort zone. Learning a new domain while simultaneously building a real product under time pressure was not easy. There were moments when I struggled, and moments when my teammates had to wait for me, but those challenges forced me to grow in ways I had never experienced before.
To bring Mind Tutor to life, I worked with a wide range of modern tools and technologies. I started by sketching initial ideas and user flows using Excalidraw, which helped me quickly visualize the structure of the product before writing any code. From those early sketches, I used Google Banana to build a moodboard and define the visual direction, helping me shape the look and feel of the product before moving into implementation. From there, I used ClaudeCode powered by the Claude API, provided by the organizers, as a core development companion to help structure logic and accelerate implementation. I combined this with Lovable Free, which allowed me to rapidly prototype interfaces and iterate in a more intuitive and creative workflow.
On the frontend side, I built the system using React and TypeScript, and styled everything with Tailwind CSS to create a clean, responsive, and scalable design. I also used Figma for quick layout thinking and validation, explored Google Stitch to better understand component composition and design systems, and experimented with tools like Framer and AI driven UI generation workflows to speed up iteration and refine interactions under tight constraints. For the AI interaction layer, we integrated the ElevenLabs API to enable real time voice to voice communication, giving the AI agent a more natural and human presence. Managing synchronization between voice input, system processing, and UI updates was one of the most technically challenging parts, but also one of the most rewarding.
What I am most proud of is, first, the courage to step outside my comfort zone and challenge myself in a completely new direction. As the saying goes, a ship is safe in harbor, but that is not what ships are built for, and this hackathon was my opportunity to step into the unknown and keep moving forward despite uncertainty. Secondly, I am incredibly proud of my teammates. They were truly amazing. They trusted me with a role that I was still growing into, and even when things became difficult, they never gave up. That trust and persistence created a strong foundation that allowed all of us to keep pushing forward together. Thirdly, I am proud of what we built with Mind Tutor. It is not just a prototype, but a reflection of our effort, our ideas, and our belief in what this product can become. Even in its current version, it already carries a sense of identity and purpose.
Through this experience, I learned that real growth comes from embracing discomfort and uncertainty. I learned how to quickly adapt to new technologies, how to collaborate effectively under pressure, and how to turn challenges into progress. I also came to understand that frontend development is not just about building interfaces, but about shaping how users feel, how they trust the system, and how they connect with it on a deeper level.
If possible, I would truly appreciate the opportunity to continue using the Claude API, Lovable, and ElevenLabs after this hackathon. I would love to keep refining and completing Mind Tutor, making it more polished and closer to the vision that we truly believe in.
Because to me, Mind Tutor is not just a project.
It is something I have poured my heart into.
And when something becomes that meaningful, you do not leave it unfinished.
You continue building it, shaping it, and helping it grow into something whole.
Thank you.

Huy Pham Nguyen Gia
Hưng Hoàng
Gia huy Nguyễn
Henry Ngo

Updates

Huy Pham Nguyen Gia started this project — Mar 21, 2026 06:55 AM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.