TippyTap was inspired by the need for more inclusive and alternative learning experiences for children who may not fully benefit from traditional screen-based education.
Many learners, especially those with different sensory or cognitive needs, engage better through movement, visual interaction, and guided participation rather than passive tapping or screen-only learning.
Instead of focusing on reward-based systems like points or levels, TippyTap explores a guided embodied learning approach, where children learn by physically interacting with space.
This approach is especially effective for:
- emotion recognition and expression
- shape and early cognitive learning
- short, focused learning sessions
- caregiver-supported learning experiences
The goal is not to replace traditional education, but to provide an additional learning modality that supports different ways children understand and interact with the world.
How We Built It
We built TippyTap as a full-stack AI-powered web application combining:
- Computer vision (hand tracking with MediaPipe)
- Backend processing (Python + Flask API)
- AI reasoning (Google Gemini 2.5 Flash)
- Voice feedback (ElevenLabs text-to-speech)
- Frontend interface (Flutter Web)
The system connects real-time gesture tracking with AI-generated feedback to create a continuous learning loop.
Challenges We Faced
One of the biggest challenges was motion tracking sensitivity.
We had to carefully adjust tracking behavior to ensure children could actually complete tasks without frustration.
Initially, we considered fully free-form interaction, but we realized that without guidance:
- users lost focus quickly
- there was no clear sense of achievement
- learning outcomes were inconsistent
So we shifted toward a guided tracing system, which helped improve:
- engagement
- focus
- confidence through completion
Another major challenge was integrating the full stack:
- connecting frontend and backend smoothly
- handling API communication between Flutter, Flask, Gemini, and ElevenLabs
- managing real-time feedback flow
For some team members, this was their first hackathon experience, so even basic tools like GitHub, API keys, and external services were new. However, this became a strong learning experience for the whole team.
What We Learned
We learned how to design an AI system that is not just technically functional, but also pedagogically meaningful.
We also learned:
- how motion-based interaction affects engagement
- how important structure is in learning systems
- how to balance creativity with usability
- how to integrate multiple AI services into one pipeline
Future Vision
Although this is a hackathon prototype, TippyTap is designed as a scalable embodied learning platform.
In the future, it can expand into:
- accessibility tools (non-verbal communication, sign language)
- rehabilitation systems
- inclusive classroom learning platforms
- multimodal AI learning environments
We believe this is just the beginning of a broader shift toward movement-based, inclusive digital learning systems.
Log in or sign up for Devpost to join the conversation.