Inspiration
We were inspired by online videos about using computer science for real-world interactivity, where physical objects and movements become part of your virtual workspace.
What it does
Flick maps gestures you make to commands on your computer. For example, pinching your fingers can minimize the current window, and you can scroll a page or open tabs with a gesture.
You teach Flick a gesture by recording example videos of yourself doing it. An AI agent then builds a recognition algorithm for that specific gesture and keeps refining it until recognition is reliable.
How we built it
We built Flick with Tauri, a Rust framework that lets us balance efficiency with a clean UI. We used MediaPipe to track hand movements from a live webcam feed.
Each gesture gets its own Python function. The function takes in gesture data and outputs whether or not that specific gesture was performed. To create a gesture, the user records:
- Positive samples: videos of themselves doing the gesture.
- Negative samples: videos of themselves doing some other motion.
The agent uses these videos to write the Python function. If the function doesn't work, it asks for more data and tries again. If it does work, it still asks for more data as a test on videos it has never seen. Because the user has just recorded them, they are genuinely new. If the function succeeds on that test, it counts as a successful gesture and becomes the function Flick uses. Otherwise, the agent uses the new data to build a better function and asks for another test later.
Challenges we ran into
Detecting when a gesture starts. It's hard to tell whether a gesture has begun at all, or whether the hands are just sitting still. Instead of trying to detect the start directly, we watch for significant movement at any point and check whether that movement matches any of the gesture functions.
Complementary gestures. Some gestures are opposites. Pinching and opening your hands do the exact opposite thing, but to end a pinch you have to open your hands, and you don't want that to trigger the "open" command. Flick lets users define complementary gestures. Whenever a gesture fires, its complement is turned off for the next 5 seconds.
Accomplishments that we're proud of
Flick is actually usable. Recognition runs with low latency, so it works for real applications like scrolling websites or Instagram Reels, or pinching to open tabs.
What we learned
We learned that building for real-time interaction is as much about handling the in-between moments as recognizing the gestures themselves. Deciding when a gesture has started, and preventing the motion that ends one gesture from triggering another, took more design work than the recognition. We also learned that testing a gesture on fresh videos the model has never seen is the best way to tell whether it will work in everyday use. And we learned that low latency is what separates a demo from a tool people can actually use.
What's next for Flick
Right now, adding a gesture means going through a full training process. We want to make that much quicker and simpler, so new users aren't pushed away when they're first trying the tool.
Built With
- anthropic
- claude
- computer-vision
- css
- docker
- grok
- html
- javascript
- machine-learning
- mediapipe
- opus
- python
- rust
- tauri
- xai
