If climbing stairs is good for us, why don’t we do it? Because we need more motivation! Through gamifying the stair climbing experience, we will help you boost your physical activity leveraging both internal and external motivation. We tap into human psychology and help you to defy our pain avoidance nature. Using positive reinforcement to reward the challenge, we turn this agonizing physical activity into an emotional and joyful experience.

What it does

The VR experiences magnifies the rewards and stimulation the user is able to get compared to incumbent mobile app competitors. This elevated gratification is also more effective to building this healthy habit and help the user to achieve their fitness goals. Externally, users' motivation will be boosted through visual effect observed by people around you. The two LED rings attached to your leg indicate whether you're using the stairs (flashing colors) and your energy expanded (progress bar) on the stairs. Internally, you'll see fun videos or encouragement to keep you entertained and empowered, then, when you reach the light at the end of the tunnel (aka. complete the stairs), you'll be celebrated by confettis!!

How we built it

Our product consists of two major components: The tracking hardware, and the VR headset display.

On the hardware side, we used an ESP32 controller board and connected some peripherals:

  • BMP388 barometer, used to detect altitude changes
  • BNO085 IMU, used to detect leg movement
  • Two I2C LED rings
  • Two vibration motors

We use the barometer to measure vertical position, calculating a difference of two moving averages (the average of the past second of measurements, against the average of the previous second of measurements). This tells us when the user is moving up or down the stairs.

To track steps, we monitor the acceleration of the user's leg using the IMU. This enables us to track their progress as they climb the flight of stairs.

As they climb the stairs, we use the LED rings and vibration motors to alert the user of their progress. One LED ring shows the activation status of the climbing mode with a color-changing glow, while another acts as a progress meter showing another light for every step climbed by the user. The vibration motor intensity increases as the user reaches the top of the staircase.

At every moment in time, the hardware emits a bluetooth message of the format "update climbing 5", indicating the state of the user and the number of steps they have climbed, or "update flat 0", indicating that the user is on flat ground.

The headset display is running a Unity application, and using Singularity to ingest these bluetooth messages. As it receives updates, it monitors for changes in the current state. In all states, a motivational video is shown using an overlay relative to the starting position of the user. When the user shifts from "flat" to "climbing", the VR headset begins tracking the steps climbed, and when the user hits at least 6 steps, confetti visuals and sound are triggered. The confetti visual is positioned relative to the stairs so the user walks into it at the top of the steps, celebrating their achievement.

Challenges we ran into

On the hardware side, we faced some issues with physical connections of the jumper wires, and the IMU sensor was frequently disconnecting. By securing the wires with zip ties and hot glue (which we ran out of towards the end), we managed to reduce the frequency of these disconnects. It was also hard to ensure a non-interrupting bluetooth connection to the headset.

We found that it wasn’t necessarily obvious how to smoothly connect and sustain a connection between an ESP32 and a Quest2 headset. Our initial setup for passthrough VR was made with OVR, nevertheless the Bluetooth UI interface was not compatible with OVR hand tracking interactions, so we had to switch to XRI adding the OVRManager script and OVR Passthrough Overlay script to the XR Rig. That solved our compatibility issues.

Accomplishments that we're proud of

In the limited time duration of the hackathon, we're proud of the length we were able to go with the hardware/VR integration, as well as step and elevation detection. On a more granular level, the custom hardware connection to Oculus VR headset was a big achievement that we are proud of. Since we were working with an under-tested library for ESP32 connection to Unity. We’re proud of producing a VR experience that does good for the environment by encouraging people to take stairs.

What we learned

We learned streaming high bandwidth data over bluetooth to the Oculus headset in passthrough mode. We explored algorithms for accurate step tracking using an IMU sensor.

What's next for ARcade Steps

During brainstorming, we had a great number of new features we wanted to include in the final version of our project, including:

  • displaying motivational pictures every time the user is making a step
  • displaying the number of steps when the user finishes the ascent
  • displaying the progress bar in the VR headset (right now, it is displayed on the hardware harness)

In the future, we'd also like to apply our technology to other types of exercises, such as running, cycling, and rock climbing. We'd also like to further develop the VR environment, such as higher quality passthrough, more immersive world building, and additional gaming components.

Built With

Share this project: