Handstand Quest: Freestyle Fitness App

Start tracking metrics for current exercise
Stop tracking current exercise and save stats
Navigate to next exercise
Pick a random exercise
Call out which exercise you want to track
Leveraging wit.ai to discover new voice commands from app users

Inspiration

I was trying to find ways to motivate myself to do handstands daily and improve the time I can hold a handstand.

To do so, I started looking for a mechanism that can help me track the length of handstand holds, with minimal effort, so I can focus on practicing rather than log keeping. Clearly, the key was that the tracking needed to happen hands free.

Gyroscope on the phone was my first idea, but then I realized it’s not that practical. I considered using a chip that attaches to the foot, but then again, it is yet another device that needs to be charged, worn, and put somewhere so it won’t get lost.

When I thought I ran out of options, I realized that using voice could be promising. It’s hands free, doesn’t require any setup, and does not force me to attach something to my body. I also wanted something without that tax of having to say “Hey Siri/Google, start my app X” every time. And voilà, I think that resulted in the first fitness app of its kind that leverages voice commands to guide a workout, one exercise at a time.

What it does

Allows a user to talk to a mobile app to guide their workout:

Start tracking an exercise by saying "start". This will either start a timer or start counting reps.
Say "stop" to stop tracking.
Say the exercise name to pick the exercise, or say next, previous, random to navigate exercises
Say "faster" or "slower" to increase/decrease the speed of reps for an exercise

Based on what users say, the app leverages wit.ai to discover new voice commands (intents).

How I built it

The app is built for iOS using the Swift language, and with firestore as the backend. It leverages wit.ai to detect intent and discover new voice commands based on how users talk to the app.

Challenges I ran into

There is no Swift SDK for wit.ai. I had plans to leverage more of the API, but decided to just focus on the messages API for intent discovery.

The voice recognition capabilities are all on the backend. It would be HUGE if there were one available only on the client side on the device. This way, the functionality would work offline and of course it will run faster. Something like pocketsphinx.

When the app speaks, it's hard to get the mic to ignore speech coming from the speaker versus that coming from the user.

Also, it would have been great if wit.ai let me explore all the messages, instead of limiting me to see 10 and instantly either removing or training them in order to see more messages.

Accomplishments that I'm proud of

Being able to recognize intents from a "continuous stream" of voice, as the user does a workout. The app focuses on recognizing between one and 5 words commands.

The app is practical and one of the very first fitness app to leverage voice for workouts without weird wake up commands.

What I learned

Leveraging wit.ai gave me a very cool idea to "start 15", which would mean that tracking starts for 15 seconds or 15 reps and then it automatically stops on its own. This came directly from a user using the app and having their command funneled to wit.ai.

Users actually use commands that are not associated with an intent, and leveraging wit.ai helps me discover new ways (voice commands) that users could leverage to interact with the app.

The voice interface needs to be carefully designed, just like UX and chatbots. There is so much potential with voice-enabled apps that have not yet been tapped into.

Voice recognition does not work every single time, mainly due to the fact that it is open-ended, and not a known set of words/sentences. This gives a ton of flexibility but reduces accuracy/confidence.

What's next for Freestyle Fitness

Go beyond exercise tracking into programs and workouts. Leverage more machine learning to check form and alignment for different types of exercises (mainly handstands).

Thanks to the power of wit.ai, allow users to ask questions like "when did I do my best handstand?", "what is my handstand record?", "remind me to workout at 6 pm" etc..