Ordinary Remote Controller

me (in a wheelchair)
3d sim
Hardware

Inspiration

As the theme was accessibility, we decided to make a project that would make daily tasks easier and simpler for people with mobility limitations. Our gestures mechanic was inspired by the google doodles halloween cat game, and we were motivated to create something that empowered people in a fun and inclusive way.

We wanted to raise awareness about the daily challenges wheelchair users face in controlling their environment. By creating an immersive simulation, we help others experience these accessibility barriers firsthand while demonstrating how smart gesture-based technology can restore independence and dignity.

What it does

Our project allows users to use a single remote with gesture controls to perform tasks. Users can turn lights on/off, open doors and windows, and control appliances throughout the home, all through gestures and hand movements.

How we built it

This project is a gesture-controlled 3D smart room visualization built with a Flask backend and Three.js frontend. The Python backend interfaces with Nintendo Joy-Con controllers via the pyjoycon library, capturing gyroscope data from the left Joy-Con for camera rotation and joystick input for movement, while the right Joy-Con tracks pointer gestures for axis-based interactions. A custom GestureHandler processes pointer movements with deadzone filtering to determine gesture axes (x/y) and values. The Flask server runs on a separate thread, exposing a REST endpoint that polls input data at 50ms intervals. On the frontend, Three.js renders a GLTF bedroom model with dynamic lighting (ceiling light, lamp, and RGB mood light), animated doors and windows, and shadow mapping. The architecture uses an analog InteractionManager system where all interactive objects (lights, door, windows, music volume, RGB channels) accept continuous 0-1 values rather than discrete states. Camera controls are handled by the camera-controls library, smoothly interpolating between Joy-Con orientation and gesture-based pitch/yaw adjustments. The entire system runs in a continuous update loop using requestAnimationFrame, with the frontend fetching controller input via periodic HTTP requests to drive both camera movement and scene interactions.

Challenges we ran into

The physics calculations requires to detect and gauge the hand gestures. Designing and implementing a heuristic algorithm to classify and detect hand gestures was quite the challenge and it took countless attempts to train and calibrate correctly.

Accomplishments that we're proud of

We're particularly proud of successfully integrating Nintendo Joy-Con controllers as intuitive input devices for smart home control. Getting the gyroscope data to smoothly translate into camera rotation while simultaneously handling joystick movement and gesture recognition was a significant technical challenge that required careful calibration. We're also proud of the gesture recognition system we built, based off just velocity and acceleration, we're able to translate that into the relative position then absolute position in order to recognize it as a gesture.

What we learned

We learned about the implementation of AI algorithms to categorize even the most insane problems. We were able to calculate the hand gestures based only on velocity and acceleration from the previous location.