What inspired me
Ever since I was a kid, I’ve been fascinated by sci-fi movies where people control computers by simply moving their hands—grabbing floating tabs, swiping mid-air, and interacting with virtual objects like magic. That dream inspired me to build something similar in real life.
With the CanMV K230 AI vision board, I created a hand gesture recognition system that can track and identify basic hand gestures in real-time. This is the first step toward creating a fully gesture-controlled interface—just like in the movies.
What I Learned
This project pushed me to explore and apply several key skills:
- Python programming: Python to prototype and test logic quickly before deploying.
- AI vision: how to work with pre-trained models, optimize them, and deploy them on edge hardware like the K230.
- Computer vision fundamentals: Understanding image processing, bounding boxes, and gesture classification.
- Hardware-software integration: Communicating between the AI model, camera input, and response logic in the board.
How I Built It
- Setup: I flashed the firmware onto the K230 board and connected its onboard camera.
- Model selection: I started with a lightweight hand detection model and customized it to focus on open palm, closed fist, and pointing gestures.
- Training and testing: I trained and tested the model using labeled hand images and tuned it to run efficiently on the board.
- Live gesture tracking: The model processes frames in real time and returns the detected hand gesture.
- Output logic: I designed simple actions triggered by each gesture (e.g., “grab” to select, “swipe” to move).
Challenges I Faced
- Limited documentation: The K230 board is powerful but not as widely documented as other platforms, so setup and debugging took extra effort.
- Model optimization: Running AI models on a resource-constrained edge device required careful optimization to keep real-time performance.
- Gesture variability: Hands come in many shapes, sizes, and orientations. Making the model robust enough for real-world use took a lot of trial and error.
What’s Next
This is only the beginning. My long-term goal is to build a fully interactive interface where users can control digital elements in mid-air—just like in science fiction. I plan to expand the gesture set, improve accuracy, and eventually integrate this with more complex systems like virtual desktops or robotics.