Fishy Business

Fishy Business Mascot (sacabambaspis)

Inspiration

We have all played classic .io games like Agar.io and Slither.io, but they all use the same control scheme, that is keyboard and mouse. So we asked ourselves: what if you could control a game using just your face? The inspiration struck when we realized that modern web browsers have powerful machine learning capabilities built right in. We wanted to push the boundaries of what is possible in browser-based gaming by combining real-time multiplayer mechanics with cutting-edge facial recognition technology. The idea of opening your mouth to boost your fish felt both hilarious and genuinely immersive and we felt that it is the kind of interaction that makes people laugh and want to try it immediately.

What it does

Fishy Business is a multiplayer fish survival game where players compete in real-time to become the biggest fish in the ocean. You control a fish swimming around eating food pellets and smaller fish to grow bigger, while avoiding larger fish that can eat you. The unique twist: you can control your fish using facial expressions and head movements detected through your webcam. Turn your head left or right to change swimming direction, and open your mouth to activate boost. Traditional keyboard controls (WASD + Space) are also available for those who prefer them. Collect powerups scattered across the map to gain unique abilities based on your fish species—like extended attack range, invulnerability shields, temporary size boosts, full map vision, or special transformations. The game features real-time multiplayer with live leaderboards, kill feeds showing who's eating who, and a minimap for navigation. We also added a Fish Racing minigame where players compete by rapidly opening and closing their mouths, with speed measured in Mouth Actions Per Minute (MAPM).

How we built it

Frontend

Next.js 14 with TypeScript for the client application
HTML5 Canvas for high-performance 2D game rendering at 60 FPS
TensorFlow.js with FaceMesh model for real-time facial landmark detection (468 points)
WebSocket API for bidirectional client-server communication
Tailwind CSS for UI styling

Backend

Go for a high-performance authoritative game server
Gorilla WebSocket library for handling concurrent connections
Custom game loop running at 60Hz with fixed timestep physics
Quadtree spatial partitioning for efficient collision detection
Interest management system that only sends nearby entities to reduce bandwidth by ~90%

Key Technical Implementations

Face tracking calculates mouth opening using vertical/horizontal distance ratio of facial landmarks
Client-side interpolation bridges the gap between 15Hz server updates and 60 FPS rendering
Species-specific hitboxes with oriented rectangles for body collision and circles for mouth detection
Single-threaded game loop with goroutine channels for safe concurrent input handling

Challenges we ran into

Face Tracking Accuracy: Initial mouth detection was unreliable - it would trigger randomly or not detect at all. We solved this by implementing a dynamic calibration system that measures each user's neutral face state and sets personalized thresholds.
Network Latency and Synchronization: With 15Hz server updates, movement felt choppy and delayed. We implemented client-side prediction and interpolation to create smooth 60 FPS motion even with lower network update rates.
Collision Detection Performance: With 300 food items and multiple players, O(n²) collision checks were killing performance. Implementing a quadtree spatial index brought collision detection from ~45% of CPU time down to ~5%.
Species-Specific Hitboxes: All fish initially had circular hitboxes, making elongated fish like swordfish unfair. We created a hitbox configuration system with custom dimensions for each of the 5 fish species.
Race Conditions: Multiple goroutines accessing world state caused data races. We adopted a single-threaded game loop model where only one goroutine owns the world state, with input sent via buffered channels.
Mouth Cycle Detection: Detecting discrete "mouth open/close cycles" for the racing game was tricky. We implemented a state machine tracking transitions with debouncing to prevent double-counting.

Accomplishments that we're proud of

Face tracking actually works! The moment someone controls their fish by moving their face for the first time is magical
Smooth multiplayer experience despite network constraints - the game feels responsive even with 15Hz updates
Performance optimization - maintaining 60 FPS with hundreds of entities through viewport culling and spatial partitioning
Species diversity - each of the 5 fish types (Swordfish, Shark, Blobfish, Pufferfish, Sacabambaspis) has unique hitbox characteristics
Complete game experience - from lobby system to real-time racing to kill feeds and leaderboards
Dual control modes - seamlessly switching between keyboard and face tracking
Creative minigame - Fish Racing using mouth movements creates a genuinely fun and novel competitive experience