Modern interior design applications are limited by the use of touchscreens. The restriction of movement is a great liability when dealing with this field of work.
What it does
Through Augmented Reality, interior designers are able to select custom furniture to place. The movement of your hands act as a real world cursor for the appliance. If you move your hand towards a certain direction, so will the object. Making a fist will allow you to lock the furniture in place, so you can move on to the next. This way interior designers and home owners alike can effortlessly try out different furniture placements when designing a room.
How we built it
Using OpenCV in Python, we take a sample of the user’s skin colour in order to differentiate it from the background. Using contours we can locate the tip of each finger and send the coordinates to the IPad using a UDP connection over the local network. Then, by calculating the movement of the hand, we generate an object with ARKit and move it accordingly.
Challenges we ran into
The main challenge was sending the coordinates of the fingertips to the IPad. We had a lot of trouble making a UDP connection between both devices because of our lack of knowledge in Swift. However, with help from mentors and collaboration of all team members, we managed to find and fix the bugs preventing this from working.
Another challenge we faced was finding a reliable finger detection program that used OpenCV. Many of them had a lot of difficulty differentiating the hand from the background and only worked in specific conditions. We ended up modifying the best working one to make it fit our needs.
What we learned
UDP Client/Server Socket in Python and Swift, OpenCV, and ARKit.
What's next for AirTouch AR
The current iteration of AirTouch AR was limited by the hardware resources available. Future adaptations will use AR Glasses to make the entire process much more seamless. We will also replace the laptop with some kind of camera to avoid having an extensive number of hardware devices necessary to run AirTouch AR.