Mirai Secured with Automation

Inspiration ✍(◔◡◔)

To Build a smart home automation system 🤖 🤖 🤖

What it does 📝 📄 ✍

The problem Mirai Secured with Automation solves 🎓 🎓 🎓 :-

Today there are millions of IoT devices but only a few address the most important issue of security and data-privacy.

Anyone who has the server access be it a hacker, or the manufacturer or the service provider itself, can control your devices.

How we built it ¯_( ͡❛ ͜ʖ ͡❛)_/¯ (̶◉͛‿◉̶) (͡° ͜ʖ ͡°)

We have implemented encryption not only at software level but also in our hardware(ESP32) by customizing the partition scheme. To implement a Server-Less model, we have Peer-to-Peer(P2P) communication between the nodes of the mesh as well as with the smartphone. To enable global access of our devices, the smartphone uploads the device status to iCloud with encryption enabled. At no time the devices are directly connected to the cloud server. .So no one can control our devices until they have gone through the initial setup process which only involves scanning of QR-code from smart-phone and all the wife and cloud credential are automatically configured. At the Hardware level, it is important to enable on device secure encryption boot and flash of the firmware. This removes the possibility of hacking into the system manually by reading the Flash memory registers of the device(ESP32).For this we have made a custom NVSpartition scheme to enable code operation modularity:

• ✿ 2 OTA update partitions for secure boot after over-the-air update of firmware.
• ✿ HomeKit and MESH running on 1 core and Wake word detection on the other.(ESP32 is a dual core SoC)

Dynamic Addition of various types of devices with just 1 product using MESH which communicates on our custom encrypted networking protocol rather than HTTPS or MQTT.

Our device doesn't require the users to download any extra app.Our device works and syncs with both Apple-HomeKit,** Google Home** apps which come Pre-Installed on all iOS and Android devices.

On Device Offline Speech Recognition

Challenges we ran into (̶◉͛‿◉̶)

1. ✔ The first challenge was to implement the mesh network. :ESP32 offers 28 sub channel state information using which we communicate with the other nodes. Setting the right channel info was a very tough challenge which we solved by calibrating each channel using the other, hence we were able to filter out the off-frequency waves and got the fundamental one.
2. ✔Second challenge was to enable local control to minimize delay when on same wifi-network. this was solved by directly relaying the messages via the wife router if using wife for communication else BLE doesn't have this problem.
3. Sometimes it may happen that your smartphone is not in the same room as your devices and you want to control them using voice commands( Otherwise voice control is enabled using Siri/Google Asst).

For that we need speech recognition system. How it is done in Alexa: They continuously send the audio captured to their server for processing. So what we did is that we enabled the wake word detection (Alexa or Marvin in our project) on the ESP32 itself. If and only if the wake word is detected, then the communication with external server(Wit.ai) starts for intent detection. Even if someone wants to control the devices directly from the server, he will not be able to do it until the wake word was detected. Hence secured from both ends. For wake word detection, LSTM is better than CNN as for less input it gives better results. It can create its own dataset if there is any error during run time. But we could only find support for CNN in TensorFlow Lite for embedded MCUs, which enables the conversion of the trained Neural Network to C++ code. On one Core of ESP32 we run the wake word detection and on the other we run the Homekit as well as Mesh network. We have also enabled users to create their own automations based on sensor data or time. Users can use the same product to control any device(Lights, Fan, Thermostat, Sprinklers) dynamically and can add multiple devices based on existing setup.

What's next for Mirai

Future Enhancements

While testing we came across an interesting concept of sensor-less motion and gesture detection based on the Channel State Information (CSI) values. ML can be used to detect the variations in the reflected radio waves between the 2 nodes as there is some motion in the room and appliances can be controlled accordingly. This can be achieved with already existing model with a future firmware update.

We will also add current measuring hall effect sensor to report power consumption and automatically control the accessories accordingly.

Built With

• applehomekit
• convolutional-neural-network(cnn)
• embedded-c
• esp-idf
• esp32
• googlecloudapi
• platformio
• tensor-flow-lite-formcu