A lot of home assistants such as Alexa, Google Nest, etc. have appeared in the market and are quite useful for home automation tasks such as switching on lights, TV, etc. But most of these assistants use a voice-first mode of communication which although convenient, have a much higher response time compared to using remote controls. What if we could have the best of both worlds? A unified control interface and a responsive user interface and experience.
This is achievable through integrating IOT and Augmented Reality. Our idea is to use the camera of the mobile to capture the camera frame and use Computer Vision with an Object Detection Model to figure out what that user is looking at and provide context-sensitive actions and information. Essentially, all you would have to do is point your phone at a light bulb and press the button on the mobile app to turn it on or off. Similarly, this would work for any smart device at home.
We propose a mobile app that uses the camera to see the world of appliances we want to control through AR. The mobile app streams the video frames to a server that processes the image frame and performs object detection on it to figure out what user is looking at and fits a bounding box around it. Then, it returns these bounding boxes back to the phone to anchor the UI content close to the physical object. This UI will be context sensitive and be dynamic based on what object the user is looking at. When we interact with the buttons in this UI, we send another request to the server, that is also doubling up as an IoT hub/broker, which sends a message to the edge device which will act according to the user input. This whole pipeline will be seamless and in the perception of the user will be as if they control their applications by just looking at them.
Log in or sign up for Devpost to join the conversation.