From Jarvis to GLaDOS, we were inspired by the fictional AIs that made the lives of our heroes so much more pleasant (if you forget about the neurotoxin). Unlike the movies, today’s smart assistants offer unnatural forms of interaction which makes it extremely painful for one control the aspects of their home. We want to create a seamless and natural smart home device flow that integrates Augmented Reality and gesture control into a centralized hub. One aspiration was a home that allows you to change the luminosity of a light bulb by pointing at it and turning your arm. We also aspired to set a group of smart home appliances to create group actions (such as a “go to bed command” to turn off your lights in living room and kitchen).
What it does
HomePoint allows homeowners to smarten up their home with natural points of interaction. The User Experience in a Nutshell:
- Set customized smart home control flows on our Google home app. (such as “set Go to bed to turn off living room and kitchen lights”). You can set as many as possible.
- Say “I am going to bed” to the Google Home. The Lutron hub will turn off the living room and kitchen lights real-time.
- Hold an AR app in front you that will identify which light you are pointing at, point your arm toward a light bulb and tilt to change its luminosity.
How we built it
Homepoint is deployed on a Node.js server that uses Firebase’s cloud functions to manage all aspects of the smart home. Through the use of Api.ai/DialogFlow, a Google Home pings our node server and relays the requests it receives to control the state of the lights. The Myo armband sends its IMU data (turning speed, correlate to rate of change in luminosity) through BLE to the laptop. In conjunction with image recognition data from the AR App, a request is sent to the server. We used a laptop to pull the server data and transfer it to the Android Things board through serial communication. The Android Things board is connected wirelessly to the Lutron hub, which would receive the signals and turn on/off lights and change its luminosity.
Challenges we ran into
Setting up NXP Android Things development board was painful because the new product lacked tutorials and documentation. Learning how to program dialog flow and google home, and integrate all components to communicate through server deployed on Firebase. Lutron board has a poor system of connecting to devices. Once we connect the laptop to the router, the laptop lost internet access because the wireless connection is occupied. Thus, we decided to connect the router with an Android Thing board and communicate input signals to Android Things through a serial port. That failed because the poor documentation of Android Things, so we have to attach a touchscreen to Android Things board and demo it manually. We were also forced to abandon our original approach with an augmented reality interface because Vuforia and OpenGL refused to display images within Unity.
We also spent time integrating a Myo Armband to change the luminosity of lights via the rotation of one’s arm. However, due to complications with reading accurate information on the arm’s position, we were unable to complete the integration with the backend server.
Accomplishments that we are proud of
We’re most proud of connecting the Lutron board to the Android things because our weekend was plagued by router issues and incompatible cables. Messed around with lots of hardware that are not designed to work together and tried our best to create integrations for them. Our picture while hacking was posted on the Daily Pennsylvanian!
What we learned
Our team could have focused more on a minimum viable product throughout the weekend. We spent too long exploring different methods of creating a cohesive smart home with devices such as the Google Home and the Myo armband. In hindsight, we should have prioritized the interaction between the NXP board and the Litron board. But hey, messing with hardware was really run, especially connecting so many hardware from different companies with different protocols. It is a good skill to practice as smart home and voice assistant are so popular and so diversified from different manufacturers, and the integration of all smart home hardware is extremely important.
What's next for Home Point
We hope to use the proper network devices to take advantage of Lutron’s wireless capabilities. A hub that integrates all kinds of smart home devices with different protocols, smart home speakers to control all your devices in one central location.