Digital Twin Instance within Warden
Isolation Tree Overlaps
Training/Validation Loss for LSTM
Training/Validation Loss for Autoencoder
Azure Digital Twin Instance
Replay of anomalous events
Solana devnet with firmware data
Manufacturing sites we support

Motivation

Internet of Things (IoT) technology has made it so that nearly everything we use, from fridges to doorbells, is accessible via the Internet. As software has become extremely accessible recently, this has created a dangerous situation where important commercial hardware can be accessed through an insecure frontend.

We bring up the recent example of 700 / 10,000 French grocery stores experiencing a cyberattack because the attackers were able to use the insecurity of software (poor passwords) in order to gain access to and control hardware in a not obviously harmful way (simply turning on the defrost, but for too many hours). It’s a poignant example of how our old security systems don’t cut it anymore and need to constantly evolve.

Our Project

We designed a hardware system that acts as a protector for breached embedded systems. It consists of a guardian microcontroller, which is not connected to the Internet, and holds the weights for an autoencoder which detects anomalies in the embedded system’s behavior.

If an anomaly is detected, the guardian microcontroller takes over, preventing the hijacked controller from doing damage. It then notifies the owner of the system about the need for immediate action, allowing for swift intervention.

This hardware control system is paired with a larger fleet dashboard, which allows clients to see the status of all of their systems.

We would like to punctuate our idea by pointing out that in 2025, manufacturing accounted for nearly 30% of cybersecurity breaches/incidents across industries (making it the most targeted industry). Much of the manufacturing industry is underprotected while the rate of bad actors targeting them specifically is seeming to proliferate, and we want to change that.

Data Collection

Our goal is to improve the security of large-scale systems, so we went in search of accurate commercial data to develop a refrigerator anomaly system.

We utilized a program called EnergyPlus from the Department of Energy alongside weather data from multiple cities to model temperature conditions. EnergyPlus allowed us to synthetically generate accurate data using scientific simulations of energy flows. Paired with preexisting historical weather data, we were able to generate anomaly data as accurately as possible.

The Details

To perform anomaly detection, we trained a lightweight architecture capable of running without WiFi on a microprocessor. We analyzed three different ML models to find the most accurate one for anomaly detection by generating a training, validation, and test set based on real world refrigerator features.

Isolation Forest We first tried to use isolation forests in which we have decision trees which randomly branch with the idea that an anomaly will take a shorter path to a leaf. Unfortunately, we were getting a 100% detection rate for anomalies and almost a 0% detection rate for non-anomalies. Essentially, the model was learning to always guess something as an anomaly. Changing the depth of the trees and the batch size didn’t seem to influence the outcome of the model, so we moved onto a deep learning architecture.

LSTM The first deep learning architecture we trained was an LSTM (long short-term memory) which is a type of RNN focused on remembering information over long series. We used pytorch’s built in LSTM architecture along with an MLP to bring the model from the hidden layer back to the feature layer. We noticed significant improvement in this model, with the test set performance being 66% in identifying anomalies and around 50% in identifying non-anomalies. This was still not good enough, so we took a look at the k most relevant features for each input and only compared those. The reasoning was that anomalies tended to affect particular sets of features at a time, so it’s better to look at the features with the highest variance than considering the aggregate influence of features. From this analysis, we noticed that recognizing anomalies went up to 86% and non anomalies to 63%.

Auto-Encoder The second and last deep learning architecture we tried was an auto encoder. This could be fruitful because if we train purely on non-anomalies, if we are provided an anomaly, the model shouldn’t know how to reconstruct it, so it will perform poorly and thus claim that we have an anomaly. Taking the top k most relevant feature for this gave us the best test data result of 98% anomaly detection with 86% non-anomaly detection. So, we decided to stick with this model when running inference.

Sponsor Tracks

SpacetimeDB

Our simulations yielded sensor and controller measurements every minute. As a result, the simulation for the month of January alone in a single refrigeration warehouse facility was nearly 13 million rows. This made Spacetime DB a critical part of our project, allowing for rapid ingestion.

Vultr

We used Vultr to take advantage of its powerful vCPUs and train our models on the 13,000,000+ entries of telemetry data we gathered across the simulated month of January 2025 (across different warehouse facilities). We created a train / test / and split set and simulated anomaly entries based on widely recorded cyberattacks.

Microsoft Azure Digital Twins

We first created a 3D model in Blender of a refrigeration warehouse based on publicly available data and dimensions from the Department of Energy. We used Three.js to demonstrate this in our web-app, but going forward as we expand the app to larger manufacturing plants we wanted to provide them an easier way to create a Digital Twin. Therefore we created a digital twin instance on Azure and added a graph representing the relationship between the machines in the refrigerating warehouse.

This graph structure would be conducive to allowing operators to manage larger and larger fleets through Warden.

Solana

We also use Solana to introduce additional security for firmware tracking by recording firmware hashes on the blockchain. Since around 48% of datacenter operating technology and around 50% of other manufacturing operating technology has old firmware that often can be exploited, a system of tracking large firmware vulnerabilities is critical. Additionally, this hash ensures that any changes to a firmware will change its hash, making sure corrupted firmware is never installed.

Armadin Our hack is built to increase the security of embedded systems and the industries that rely on these systems most.

Challenges

Training the model proved to be more difficult than we realized. There are many factors which can influence sensor outputs, most notably seasons!

To see how generalizable our model is to anomalies, we tried testing it on real world data from different times of the year. This caused us to receive a lot of false positives, which puzzled us because from our testing on January data, the model performed really well. However, because our train/test/val set was only from January rather than from the whole year, our model did not account for weather as well as expected and we detected larger numbers of false positives.

This, combined with the difficulty of choosing an accurate and verifiable simulation, made model generation an unexpected challenge.

Future Goals

We plan to extend our system to commercial manufacturers, factories, and food and pharmaceutical lines. We’ve specifically focused on these areas because they lack the critical safety fallbacks that certain chemical plants and other such locations have.

To accomplish this, we plan to create a more pipelined approach to training models (since different manufacturing plants would operate under different conditions and require different models). For devices in similar environments, we can reuse the same model weights, but since we need to be sensitive to noise, we are required to retrain the same architecture.

We also aim to create a more synchronized view for the maintainer to analyze patterns in attacks across time and space. Since SpacetimeDB easily keeps track of historical data, and digital twins allow clients to visualize the effects of hacks on their systems, we plan to incorporate informed hack scenarios. This provides the user valuable insight on the impact different types of hacks can have on their system, enabling them to implement preventative measures.