LivePresence

Inspiration

At a previous hackathon, one of our friends noticed that it was extremely common to see projects involving the tracking of crime in neighborhoods, but so far, none of us have seen a project that helps people after an intrusion has occurred. This is where LivePresence comes in. Instead of providing data on what’s surrounding a home, it provides information straight to the user on what is happening within their home. Through a modular ensemble of sensors, LivePresence is able to provide information straight to a homeowner regarding how their home is currently, even in benign conditions, but also after someone may have entered. Furthermore, the user is encouraged to directly interact with the sensors using the chatbot feature implemented. Additionally, a secondary feature of this project is that it can help homeowners improve home power efficiency by informing the homeowner of times when there is little to no activity in the home, allowing them to turn off HVAC or other high-energy usage appliances.

What it does

LivePresence is a system that collects, displays, and analyzes data from various sensors, including CO2, temperature, motion, humidity, and Volatile Organic Compounds (VOCs). This data is passed into a hybrid quantum-classical machine learning model to determine the presence in the area. This quantum machine learning model gives us the advantage of being able to represent more states with fewer classical model parameters, saving memory and computational costs. The information from the sensors, model output, as well as an OpenAI prompt, are displayed to a web app presented to the user. The prompt allows access to a GPT 3.5 model powered by the OpenAI API, with a premade prompt to give the model context as to the information it should be describing.

How we built it

The core of the input is an ESP32-WROVER microcontroller, which ran an MQTT publisher to a Mosquitto server running on a smartphone. Data from this server is then pulled by a Flask web app running on a laptop, which updates the UI and runs the assorted data analysis algorithms, including a hybrid quantum-classical deep learning algorithm to determine occupancy from the sensor data. This UI was built using typescript shows the sensor data, and deep learning model outputs, and includes a GPT-3.5 chatbot interface with a prompt to give outputs as if it were a home security consultant. This is powered by the OpenAI API, which is also running as a backend Flask server. The hybrid quantum-classical deep learning algorithm was written using PyTorch and Qiskit, with a PyTorch neural network for the classical part and a Qiskit quantum circuit at the end for the addition of quantum algorithms into the network.

Challenges we ran into

One of the hardest parts about this project was getting the front end to reliably work, as it would occasionally cause issues with the backend and the MQTT data. We also had issues with the OpenAI API, as with the initial integration of the API we kept getting 500 errors. This ended up being due to some issues with formatting the output of the API request, which we eventually solved. The hybrid quantum deep learning algorithm also caused some trouble, as it was difficult to train due to inconsistencies in GPU support. Classical models are easily trainable on GPUs, and modern frameworks such as Qiskit also include GPU backends, but they use different APIs and drivers for each of them. This made it difficult to train on GPU, as each switch between PyTorch and Qiskit necessitated moving the data back to the CPU, passing it onto the other framework, and then having that framework move it to the GPU again. This communication overhead was so severe that it ended up being more than twice as fast just training on CPU, which is what we ended up doing. This still took a long time to train, with a training time of around 5 hours.

Accomplishments that we're proud of

There are many accomplishments in this project that we’re part of, one of which is dividing and conquering this project into smaller, more easily solvable subproblems. Each subproblem is then solved recursively. We have also successfully implemented various sensors including CO2, temperature, motion, VOCs into our system. Additionally, we drifted off from traditional machine learning methods with a hybrid quantum deep learning algorithm with PyTorch and Qiskit. We also managed to create a chatbot interface amidst obstacles in the front end.

What we learned

We learned a lot about modern technologies involving quantum machine learning algorithms, as well as the combination of systems with AI, embedded programming, and web programming. This system involves a complex set of services that must all work together to function properly, including an MQTT broker running on a phone with the Mosquitto service, an ESP32 device to read and forward data from the sensors, an MQTT client on a laptop, the web app itself (including its frontend and backend components), a machine learning backend, and an OpenAI API backend. Orchestrating all of these services can cause a lot of integration issues, and we learned a lot about the integration of multiple components into a single service.

What's next?

Firstly, it is definitely expected that we will package the hardware side of things into a commercially viable product. This project will have more capabilities, including the ability to turn off HVAC, lights, and other climate control services when presence is not detected, in order to save power. Then we would like to hopefully ship our project into a commercial environment from which, we can see real improvements in time to respond to potential home intrusions and more.

Additional credits and references

Wang, Qi et al. “Fear and Logging in the Internet of Things.” Network and Distributed System Security Symposium (2018).

Andrew C. Su - University of North Carolina at Chapel Hill (Helped with sourcing initial machine learning model) Colton Fox - Virginia Tech (Front end assistance) Dexi Lin - Georgia Institute of Technology (Hardware assistance)

Built With

• openai
• python
• pytorch
• qiskit
• typescript