How the computer sees each of our Tyles
Have you ever seen a scifi movie or any movie in general with a cool digital dashboard floating in the air? Have you ever wanted to use your hands to fling open a window? Well we have and we wanted to remake that. Unfortunately, as poor college students, we couldn't afford the tech to have the dashboard be floating in air. So we used the next best thing, paper and cardboard!
The Longer Pitch
Tyle is a customizable paper dashboard that allows you to control your computer through the use of cardboard tiles. Want to open Chrome? Throw the Chrome Tyle onto the dashboard. Want to DJ some sweet music for your party? Throw the Spotify Tyle onto the dashboard and mix up the sound with the volume and play Tyles. Need to write a paper but you don't want to type it? Throw the Microsoft Word and dictation Tyles onto the dashboard and rev up your next masterpiece. It's as easy as that. Gone are the days of managing hundreds of applications on a crowded desktop. Leverage the power of having 21st century technology at your fingertips today. Compute in style, use Tyle.
What it Does
Tyle currently allows users to open Chrome, Word and Spotify. Within Chrome, users can navigate to different sites using the search Tyle. Within Word, users can use the dictate Tyle to transcribe their speech into text. Within Spotify, users can control the flow of their music with the play Tyle. Across all applications, the volume Tyle can be moved up and down the dashboard to adjust the system volume.
How we built it
We use a webcam to capture a live feed of our paper dashboard. Building on some simple OpenCV functions, we designed a vision system to recognize and handle each individual Tyle, each of which is distinctly colored so that it can easily be recognized. Each Tyle is mapped to a different functionality, whether it is opening an application, or performing a specific action within the current open application. For instance, when the Spotify Tyle is on the dashboard, the play Tyle will play or pause the music. The volume Tyle works across all applications and allows the user to raise or lower the volume depending on the position of the Tyle. We also support both voice input in the form of transcription and handwritten URL navigation taking advantage of Microsoft Cognitive Services handwriting API.
Challenges we ran into
Ensuring that the Tyles could all be identified correctly in different lighting conditions was an extremely difficult challenge. In the end we used k-means clustering to get the primary color of each Tyle, which is then mapped to the closest defined Tyle color. However, depending on the lighting conditions, certain colors could still easily be mistaken for other colors. Further issues arose in choosing Tyle colors that would be distinct enough from each other that our program would not confuse one Tyle for another. Lighting is the worst thing to be trying to stabilize at 5am in the morning when everything else works. After much experimentation, we found pairing a secondary color to each Tyle corresponding to the color at its center allowed us to uniquely identify Tyles across all sorts of lighting conditions.
Designing the flow of the end user experience such that it felt both natural and fun to use. Originally we had the dashboard split off into different sections, with the top left corner being reserved for opening applications, the entire right side exclusively for the volume Tyle and the rest for functions. After gathering feedback from our fellow hackers we realized that this felt limiting and did not accurately capture the feeling of freedom conveyed by a customizable dashboard. This led to us eliminating the sections and allowing users to place the Tyles anywhere on the dashboard – thus properly capturing the fantasy presented in scifi movies.
Accomplishments that we're proud of
- Being able to identify the different Tyles in different lighting conditions.
- There were many different parts to this project, each action Tyle had its own backend challenges that needed to be figured out and then integrated into the overall application. Being able to create these standalone features and seeing them come together to form a great end product was nice.
What we learned
- A lot about making system calls with Python.
- How to use k-means clustering to figure out primary color.
- Drawing out important information from simple images by taking advantage of OpenCV primitives and designing some of our own innovative algorithms.
What's next for Tyle
More application Tyles and action Tyles. Currently we are limited by the background color of each Tyle, but with more time we could tailor our code to recognize the logos of each application thus allowing for unlimited application and action Tyle support.