Inspiration
We have seen many families install home theatre systems into their houses. But, most of them are unable to utilize the full capability of their speaker systems because they don't know the best positions to place their speakers at. We decided to solve this problem by making a website that people can run on what they use most i.e. their phones, such that it provides accurate acoustical analysis of a room from its 3D model.
What it does
Our product comes as a hardware-software combo package that scans a room, obtains its 3D point-cloud data, it, computes sound distribution and reverberation times, and finally represents the room on the website as an interactive 3D visual!
How we built it
The project consists of three major parts, namely - room scanning, acoustic analysis and website rendering.
1] For the first part, we connected the DC MTP8450 with Snapdragon HDK8450 and collect a 2d video (without LiDAR). The video is then broken down into frames and key features are extracted from each frame. These key features are used to match adjacent frames and calculate the rotational and translation matrix of the camera movement. We then use these matrices combined with the intrinsic matrix of the camera to triangulate the 3D points. We then stitch these 3D points to build a point cloud file.
2] Next, we pre-process the point-cloud data by performing triangulation to obtain the surfaces/walls in the room. After that is done, we run the ray tracing algorithm on the room to obtain the frequency of sound that a listener will hear at a specified location. This algorithm takes into account multiple reflections of sound waves on room walls. It also calculate reverberation time for the room.
3] Finally, all the above data and results from the analysis are sent via an API to the website that renders an interactive 3D model of the room, and presents the results of the acoustic analysis to the user.
Challenges we ran into
Going through the project part-wise, following are some of the challenges we have faced:
1] We were unable to install python and necessary packages for our algorithms to run on the HDK because of tight security permissions on the HDK.
2] Despite best efforts, the accuracy of triangulation isn't high enough and because of the approximate nature of acoustic calculations, it leads to a slight reduction in precision.
3] Because of time constraints, we were unable to finish our React Native app and instead built a website. Moreover, we were unable to host our backend API that connects each working part of the product.
Accomplishments that we're proud of
This being the first time our team has worked with HDK and integrating code and websites on an HDK, we were happy when we were able to first set up the HDK with python3 and other packages. Our first break through was being able to generate a point cloud file from a 2D video. The next major accomplishment was to successfully implement the ray-tracing algorithm for sound waves and predicting the output sound value accurately. Finally, we were able to accumulate all simulations and results to be displayed on our website.
What we learned
Through this hackathon, we learned the importance of thorough planning and careful execution of the plan. Moreover, we understood that each member in a team has their own skill set that must be taken into account while planning out a project. If both of these are followed, then the project is bound to turn out good.
What's next for room-tone
Next, we plan to speed up the room scanning and point-cloud data-processing algorithm. We also plan to implement a more realistic sound model that considers attenuation and multiple sound sources interfering together at the same time in the ray-tracing algorithm. Finally, we plan to make a more fully-featured website including authentication and cloud storage, so that professionals could use this and store their acoustic analyses online
Log in or sign up for Devpost to join the conversation.