Street Striker

Inspiration

Inspired by the energetic feel of Wii Sports Boxing, we wanted to bring that same physical thrill to other fighting games. Traditional arcade titles usually confine combat to button taps and joystick flicks, so we set out to break that mold. By fusing the intuitive motion of Wii Sports with the depth and competitive pacing of an arcade fighter, our system turns real-world punches and footwork into in-game actions-letting players feel the physical weight, timing, and adrenaline of every round.

What it does

We developed a real-time motion-control system that transforms full-body movement into an arcade controller for Street Fighter 6. A camera tracks player posture for movement: Leaning to advance or retreat, crouching, and jumping. Additionally, wearable gloves with sensors capture physical strikes, mapping left punches to light attacks and right punches to heavy attacks. The system translates these actions directly into standard gamepad commands, letting players control the fight without touching a traditional controller.

How we built it

We built the system around a multi-modal control pipeline that synchronizes three distinct input streams: computer vision, inertial motion units (IMUs), and live in-game data. Real-time computer vision tracks the player’s spatial posture, using keypoint displacement and torso angles to execute continuous locomotion like walking, blocking, crouching, and jumping. Simultaneously, wearable gloves with IMUs (MPU-6050s running on an ESP32) capture high-frequency acceleration spikes to instantly detect strike impulses, differentiating fast left jabs from heavy right crosses with minimal latency. Finally, we used in-game telemetry to create a closed feedback loop-reading game states, animation frames, and round status to dynamically calibrate input sensitivity, debounce false triggers, and ensure real-world actions align perfectly with the game’s internal timing windows.

Challenges we ran into

A major hurdle was strike classification. Specifically, distinguishing attack intensity so a heavy right cross wouldn't misfire as a light punch. Because inertial sensors measure raw acceleration spikes, a snappy flick of the wrist can generate an initial peak comparable to a full-force power strike. Rather than relying on unreliable force thresholds, we redesigned our mapping around distinct physical trajectory planes: horizontal straight punches map to light attacks, while vertical upward acceleration triggers heavy uppercuts. Leveraging directional axes rather than raw magnitude made attack detection significantly more reliable and intuitive.

Accomplishments that we're proud of

We are proud of successfully bridging hardware and computer vision into a synchronized, real-time pipeline to build a game controller. Bringing physical motion control to a game as demanding and iconic as Street Fighter was challenging, but seeing our real-world punches and movement translate seamlessly into competitive gameplay made it worth it.

What we learned

Building this system gave us a deep appreciation for the fine line between physical movement and digital responsiveness. We learned that raw sensor data is messy: a fast flick of the wrist can look nearly identical to a full-power strike on an accelerometer, forcing us to dive deep into signal filtering and dynamic thresholding to reliably classify attacks. We also learned how crucial latency optimization is when designing an interface for a competitive game like Street Fighter-balancing high-frequency ESP32 telemetry with computer vision processing taught us how to fuse completely different hardware and software layers into one cohesive, low-latency pipeline.

What's next for Street Striker

Next, we want to integrate voice-controlled Super Arts using the ElevenLabs Realtime Speech to Text API. In Street Fighter, Super Arts demand complex multi-directional motions like double quarter-circles plus a punch. Pulling that off with whole-body movement in the middle of a fast match can feel clunky. By opening an ultra-low-latency WebSocket stream via ElevenLabs STT, we can transcribe live microphone audio in real time with low latency. When the stream catches iconic callouts like "Shinku Hadoken!" or "Level 3!", our backend will immediately inject the exact input macro, giving players a way to unleash their Super Arts without losing their physical rhythm.