SWISH (Serial Wireless Interactive System for Humans)

The main interface
Calibration sequence
Gesture customization
Adding new gestures/voice commands
Gesture tutorials
Settings

Inspiration

We were inspired by a previous Hack the North project that turned a nerf gun into an FPS controller, as well as controller-free VR technology. We wanted to make things more accessible, so we settled on using something that everyone has, their hands.

What it does

SWISH is a Windows app that allows users to fully control their mouse and keyboard by using hand gestures. A webcam tracks your hands to drive the mouse and hold keys, and your voice triggers keystrokes and macros. Both run live, with latency low enough to play real-time games like Minecraft, Rocket League, Aimlabs, and Bloons TD 6.

How we built it

Our project is a .NET solution split into 3 C# projects:

SWISH.App is a WPF shell (.NET 10, Windows) with a system tray icon. It hosts both input engines and adds a styled UI for calibration, a controls overview, settings, a tutorial, and a command editor.
HandGestureRecognition (.NET 8) contains the GestureEngine, which turns camera input into mouse and keyboard input.
VoiceKeys contains the VoiceEngine, which turns microphone input into keystrokes

Hand tracking is done via OpenCvSharp4 for capture and image processing and ONNX Runtime for inference, and ElevenLabs API is used for custom voice commands that can be mapped to keybinds or macros.

Challenges we ran into

The main challenges we faced were working with .NET, as even though our two developers have some experience with it, there were still plenty of things we needed to troubleshoot. Another challenge was merging work, as our two main systems were initially independent of each other.

Accomplishments that we're proud of

We've managed to create a really unique project in 24 hours that we think many people will enjoy.

What we learned

Some things we learned include applications of computer vision and speech-to-text technology, as well as working in a multidisciplinary team.

What's next for SWISH (Serial Wireless Interactive System for Humans)

Refinement and optimization, especially with our API usage.

Built With

.net
c#
elevenlabsapi
onnx
opencvsharp4

Submitted to

StormHacks 2026

Created by

Voice to input. I connected ElevenLabs realtime speech-to-text to the VoiceEngine through a WebSocket.

UI/UX. Heather and Noah designed the user experience. I turned their wireframes and mockups into the WPF app. Together, we agreed on which features we could build in one weekend. I then built the design system, the calibration flow, the Controls screen, the command editor, and Settings.

Integration. I moved the gesture frame loop out of the console program and into a hostable GestureEngine class. The SWISH.App process hosts this engine on a dedicated thread, and the VoiceEngine runs asynchronously beside it.

Performance. I made certain performance optimizations for ONNXRuntime to prevent it from overutilizing cores while idling and in general.

Calibration. I built a 30-second calibration, which allows the app to collect hundreds of samples and corrects for how far the hand lags behind the ball.

Custom gestures and commands. The user records a pose one time, and SWISH recognizes it from different angles and distances. The command editor connects a gesture or a phrase to a list of key presses.

Polish. I also made the animated splash screen, rebindable gestures, the tutorial overlay, and the project landing page.

Krish Tiwari
I worked on developing the hand gesture recognition system for mapping hand gestures to mouse and keyboard inputs, as well as quality assurance for the application as a whole.

Arron Roasa
Heather Nguyen
Noah Ng