Harmony.io( Team number-5, Team name -!404Found)

Cover
Albums
Music
Demo

Inspiration

Harmony.io is built with the aim of bringing people together from different parts of the world on a common platform where they can have fun, connect with people, and compose music at the same time. As we all know, the Covid-19 had a drastic effect on all of the music industries. At this very moment, it is hard to have a good social life, and moreover, people are suffering from mental illness and disorders like anxiety and depression and we believe that Harmony.io can really help people having these problems.

What it does

Harmony.io is a friend for pandemic and lockdown that uses concept or AR to implement gesture-controlled instruments like Piano, Drums, Guitar, etc in an OpenCV environment. So all one needs to have is a laptop with a webcam and you are good to compose songs alone or along with your friends as a team and it can also be used as a virtual meeting platform to socialize and also hear to all the songs composed by people all over the world and also add your songs on platforms like Spotify, Amazon Music, etc. We also use image segmentation deep labv3 model to provide a virtual background option to preserve privacy because obviously, we don't want people to see how your room looks like. There is also an option to reach out to us incase of any queries or suggestions.

How I built it

The major component of our software is the video streaming feature which is built with Agora. Our main implementation is based on the gesture-controlled feature using which you can play the instruments just by the actual gestures that we use for playing particular instruments that are done very efficiently using OpenCV. There is also an option to change the background to have an effect of a real band which is done using the deeplab_v3 segmentation model. The entire frontend is built in HTML/CSS and all the algorithms and the AI implementations are done in Python

Challenges I ran into

1) Improving video quality of OpenCV/webcam. 2) Implementation of the virtual background system 3) Integration of frontend to agora.io engine 4) AI-based autotune system

Accomplishments that I'm proud of

We were able to integrate all the models with our front end to build this application and also the video calling functionality. All our teammates learned a lot of new things from this project which we had never built till now and sitting the whole night discussing how to make things work was actually a big task especially when we were working remotely and is also an accomplishment that our entire team is proud of.

What I learned

1) Computer vision and Artificial Intelligence and Deep Learning algorithms like human segmentation 2) Audio Signal Processing and Tuning for the efficient transfer of sound. 3) Learned how to implement Camera X in our application to get the frames.

What's next for Harmony.io

Right now our platform works for a one-to-one call system. We plan to extend it to the Group of people as well. We're planning to add more instruments like flute, saxophone, guitar, and violin. For the beginners, we will be providing courses through which they can learn these instruments, socialize and have fun which will help them overcome the pandemic and mental health problems like anxiety and depression.