Inspiration

Whistling is a fun thing to do, so we decided to make it useful.

What it does

Blowboard is a proof-of-concept of a USB device that allows one to whistle to use their keyboard hands-free. First, it detects a face using the computer's webcam, then starts polling for audio from a microphone. Then, it analyzes the audio, and based on pitch changes emulates keyboard input.

How we built it

Blowboard was primarily built using Python-Librosa, Python-numpy, and Python-pandas. Hardware was made with Arduino, breadboards and an electret microphone.

Dataflow

User stands in front of webcam → Blowboard starts polling for whistles → Blowboard hears a whistle → Blowboard simulates a pre-defined keypress → AutoHotKey runs the macro attached to that key being pressed → Blowboard magic!

Challenges we ran into

  1. Not understanding Python-Librosa's functions properly
  2. Figuring out how to set a threshold for whistling after getting all of the data
  3. Realizing that Raspberry Pi's don't support audio in at all through audio jack
  4. Getting the microphone to operate properly on a breadboard with an Arduino Nano
  5. TensorFlow doesn't support the latest version of Python

Accomplishments that we're proud of

  1. Being able to analyze live audio and separate it into distinct pitch sections
  2. Simulating keyboard keypresses using arduino

What we learned

  1. Python-Librosa is very good for analyzing audio from streams and existing files
  2. Mapping pitch changes to keystrokes is much harder than just mapping frequencies
  3. Raspberry Pi's don't support audio in at all through audio jack
  4. Learned about FFT for spectral audio analysis
  5. Learned OpenCV for face detection

What's next for Blowboard

Implement the arduino/raspberry pi microphone setup to complete the USB device, as currently it only runs on our Windows computers due to being unable to find parts

Built With

Share this project:

Updates