HouseCat iHome Controller

Inspiration

One of our group members got hit by a bus and couldn't walk for a while, and was left wanting a way to turn his lights on and off without needing to get up and walk to the light switch. From that experience, we came up with the idea to integrate accessible controls and wearable technology with smart home technology, to allow users with restricted mobility (or who just want to feel like a wizard turning their thermostat up with a flick of the wrist!) to control their surroundings in a new way.

What it does

Our project is a wearable ring that uses an accelerometer to track the user's movements and a machine learning algorithm to determine whether their movement is one they have set to perform a function in their smart home. For example, the user may decide to assign the act of raising their arm to turning on their bedroom lights, or making a circle to increase the temperature of their thermostat. The accelerometer data is sent to a computer (which will in the future be a centralized smart home hub), which uses a machine learning algorithm to classify the movement as one of the user-set functions or as none of those functions.

How we built it

Physical Frame

We started with a reasonable shape for something that would comfortably fit on an arm, considering that the prototype would be substantially larger than our final project. We continued to shrink the project down, lessening the footprint. The inspiration of the design was Star Trek-esque sci-fi and Marvel-style superheroes, keeping in line with StarkHacks' homage to Iron Man. One of the physical frame-building challenges was accommodating the slower-to-prototype electronics with each iteration.

Electronics

We started by figuring out how each component will communicate, prototyping with a breadboard. As we proved our design, we transitioned to PCB design to minimize footprint. Splitting into two PCBs - one for measurement and one for communication - allowed a very unique design. Each component's footprint was carefully modified to minimize PCB printing time.

Communication Protocol

We started with the most simple problem in our communication protocol - interfacing our IMU with our microcontroller. We planned our I2C connections and interfacing, which worked relatively quickly. Next, we had to figure out WiFi. Espressif provided help and examples for us to get the board connected to the internet. We had to learn how to set up our own server to receive our data and send it for processing, and ultimately send a decision to the smart home framework to execute some action.

Classification Model

We wanted to essentially use a clustering model that used each target class's data point as the center of the cluster. We tried implementing this using wavelet features, embeddings, and both combined for the preprocessing strategy, combined with a centroid model, Gaussian model, and one-class SVM for the classification model. The best performing combination was the wavelet features only with the centroid classifier, so we chose this model for our classification. We gathered training data for a few different classes of movements, as well as some unclassified movement for semisupervised learning of the representation of the movement vectors. We trained a model on this data, saved the parameters, and used it on the server to classify each movement.

Challenges we ran into

We struggled to find an appropriate power source that would supply the mechanism with enough power to function without overloading it and while remaining small enough to fit on the user's arm. While it is certainly not the end-goal, we realized that a portable charger that one of us brought was able to provide the perfect amount of voltage necessary, while fitting comfortably on the arm. This allowed us to provide consistent power to the mechanism without causing a major inconvenience to the user.

Scope management was also quite difficult, as we quickly realized that our goals were not all achievable in the short timeframe we had. We came into the hackathon with only a vague idea that we wanted to create a wearable gesture-control device for a smart home, without much of a plan beyond that. We hoped to get it to a point where we could actually demo turning on a smart lightbulb with the project, but realized that the additional work required to set up the connection between the smart bulb and the wearable was simply not feasible in the time frame available.

Figuring out the proper models to use to properly frame the electronics was also quite a feat, as we went through many iterations of designs, including multiple prints. Managing so many changing plans quickly became a challenge, especially as the electronic configuration changed in tandem with the housing needed to contain it.

Finally, the machine learning aspect was quite difficult to nail down. We did not have a dataset, so we had to generate our own data, and couldn't even start this process until the electronics were hooked up. So, the first portion of the time was spent trying to brainstorm good approaches, without the ability to test them on realistic data. In the end, we developed three different pre-processing models and three different classification models, testing each combination to see which worked best with the data. However, the process of getting these models to work was challenging, especially while lacking data for the initial portion of the process.

Accomplishments that we're proud of

Our greatest accomplishment was probably our teamwork and project management. Each of us had a role, playing into our own strengths while still remembering to have fun and learn from the experience. One member worked on the 3D printing, prototyping, designing, and printing the structure of the wearable. Another member worked on the electronics, piecing together each component and printing a cohesive set of circuit boards. Another worked on the interface between the electronics and the ML pipeline, working with the ESP-32 protocol and WiFi setup to allow the wearable to wirelessly communicate with the central computer. The last member worked on the ML model, planning the data structures necessary and the most effective models, developing the preprocessing pipeline, implementing the models, and training the models.

What we learned

We definitely learned a lot about working in a team on an engineering project. While we've all worked on plenty of group projects before, the extremely quick turnaround of this experience added another layer, as we had to keep clear communication without any time to waste.

We also learned a lot about what we can achieve when we don't expect perfection. Like many engineering students, we are generally high performers in our work, and often expect a lot from ourselves. Going into a project with the expectation that we will not end up with a finished product was both refreshing and challenging, as it required a real shift in our mindset as we approached the task. We had to prioritize which ideas were reasonable to try to implement in our limited timeframe, while understanding that the project as a whole would not be complete to our usual standards.

What's next for HouseCat iHome Controller

We can integrate finger controls through the use of rings, allowing more dexterity and finer controls. With more time and more materials, we could move from the dev board to a flexible PCB and acquire smaller versions of the electronic components in use to optimize our space.

We can connect the device to a smart home hub using the Rainmaker smart home API system. Using ESP-32, we can interface directly with Rainmaker for customizable smart home applications.

We can roll out an app that allows users to customize which gestures control which smart devices in their home, even allowing users to make their own gestures and assign actions to them.

We can train a foundational model for gesture recognition that will improve the accuracy of gesture classification and facilitate the use of custom gestures in few-shot unsupervised learning, not requiring many examples for each class.