SweetSpot

Inspiration

One of our group mates (Aaditya) had struggled with his family on what kind of home theater setup he and his family should have. After doing some research, we found that other people struggled with the same exact problem. Planning and design services cost anywhere from $500 - $3,500. Spending costs between $1,000 - $15,000+. This doesn't even account for the month spent doing these. It's not a cheap world.

So what we decided is to answer the question of, "how can we make this whole process a lot cheaper, and easy for the user?"

What it does

SweetSpot uses the LiDAR sensor on your phone to take a scan of the room. Once it does so, it asks some quick questions on what your budget is and what kind of home theater setup you would like (5.1, 7.2, etc). Also taking pictures of your room so we can determine the materials (we account for those too in our optimization!) From there, our app takes it away and generates a 3D view on your laptop on where the optimal speaker, projector and screen placements are, and an AR view on your phone so you can see what your home theater looks like in front of you before you even start buying. All staying within your budget and giving recommendations based upon the range.

How we built it

We utilized AR kit in Swift to utilize the LiDAR and AR features during iOS mobile app development. Using React, TypeScript, Vite, and Three.js we are able to generate the 3D model on the laptop. A Vultr backend is able to integrate and host all the components together.

Gemini is used for the material identification when the user takes pictures of their walls and ceiling. We made our own in-house shoebox image-source solver for the acoustic physics, the library pyroomacoustics is utilized for validation on the result. PyYAML loads in the YAML files for screen and projector specifications.

We utilize JSON room files and generated 3D equipment models to be able to load previously scanned rooms for ease of storage and access. SQLite is used storing the devices and their specifications. SerpAPI is used for searching the web for device specifications and their price that can then be used for recommendations to the user. This uses a fallback in case of a database failure as well.

Challenges we ran into

We struggled with getting the physics to work right initially, however, we found out that image sourcing methods could lead us to determine how the acoustics are simulated across a 3D room.

Furthermore, getting the camera controls and projection previews to behave correctly across different views was a massive problem. We found ourselves having a black screen sometimes due to incompatibility or lack of view management.

Furthermore, backend configuration was an issue as we struggled to make our endpoints correctly and host the Vultr server right.

Accomplishments that we're proud of

The main accomplishment that we are proud of is that our app actually does everything it says it does. We are able to select an object to reveal its description in the scene, the backend works well, and the physics is correctly validated and determined to find the optimized home theater layout.

Furthermore, we are proud of the AR features of this project, being able to walk around in a room to see how it look from your point of view is something that is a massive immersion boost for any user.

What we learned

The main things we learnt were how to orchestrate such a large pipeline and make complicated backends abstracted away from the user. We also learnt a lot about how the physics behind acoustic simulations work (and making them efficient), camera controls, and clear configuration data as we had to pass clean data throughout the pipeline for maximum clarity. UI/UX development was also a plus as we had to determine what looked simple and appealing for the user.

We also learnt a lot about mobile app development as we delved into that for the hackathon to combine both the hardware of the phone and software!

What's next for SweetSpot

We want to be able to support more home theater configurations and richer equipment models. We would also like to determine how we can make the physics optimization and Gemini integration faster in order to delay response time to the user.

Lastly, we also want to show budget constrained local vendors that the user can go to in order to achieve their ideal home theater setup and get it quick. All while accounting for the quality of their work as well.