Inspiration: The story is that one evening our group kept reading about the local transformer fires, news of EV charging points failing, all with the same old story. 'Undetected overheating', 'failed days before it was discovered.' Someone really had to be watching for it, and of course no one was watching - because while there exist thermal cameras for these kind of problems, they cost anywhere in the hundreds of thousands and are only within reach of the huge, big time utility companies. It seemed criminal that the many thousands of transformers, signal cabinets and water pumps actually run a city just didn't have this level of vigilance. We had never aimed to build a 'smart city buzzword project' but rather to answer one really simple question.
That question:
could we replicate, nearly, the capability of a ₹200,000 thermal camera with the AI on top of a ₹2,000 thermal sensor? That bet is essentially the idea behind ThermalTwin.
What it does:
ThermalTwin monitors the health of equipment through the lens of temperature - whether that equipment be a transformer, an EV charger, a traffic signal box or a water pump - all via the inexpensive MLX90640 thermal sensor with an ESP32. But rather than simply setting alarms when an asset goes above a specific temperature, it creates a 'digital twin' of what constitutes 'normal' thermal behaviour for that specific asset. This way it is able to detect anomalous behaviour before it reaches a danger point. Once it is convinced something is going wrong, it has the potential to trigger an automated action, such as engaging a cooling fan, as well as triggering a notification so someone can monitor the overall health of a city infrastructure without having to discover equipment failures.
How we're building it:
The work can be divided loosely into three main categories - the hardware, the brains and the application. - The hardware: ESP32 which is connected to the MLX90640 thermal camera, as well as the relay and the fan (so the system can do more than just sound an alarm). - Making the £200,000 camera redundant: the resolution on the MLX90640 is pretty poor in comparison to the kind you'd see in any research facility, so we are implementing a U-Net over the raw camera frames in order to boost the quality before feeding this into other models. - The digital 'twin': An Autoencoder will work to learn the 'normal' behaviour profile of any asset, which will differ between the various kinds of infrastructure - no single model will be able to describe the temperature dynamics of a transformer and traffic cabinet for example.
Predicting future failures: an LSTM model to look ahead in time to predict when temperature is going to increase, and an isolation forest model to double check for any immediate, anomalous spikes. - The plumbing: a REST API in FastAPI, a React application in Next. Js for the user interface, Postgre (Prisma) as our data store, and MQTT and Firebase to help with data flow between the device and the back end, while Twilio handles the notification sending. At present we have been concentrating on getting the core components working well, but we are still in the process of having all of the separate systems talking to one another efficiently.
Challenges we're running into: At present the single largest obstacle we are facing has not necessarily been a specific model, but rather having them talk to each other. ' Our current stance is that neither can be trusted to command physical equipment such as the fan without the consensus of both systems; after all, having the fan turn on and off at random isn't helping anyone. ' Thus, for our Digital Twin, we're looking at using a system based on learning 'normal' patterns, rather than labeling and learning from known failure events, which aren't readily available. Finally, running four separate models (U-Net, Autoencoder, LSTM and Isolation Forest) without overheating the edge device itself is another piece of the puzzle we are still working out, in terms of what can and should be done by the device vs what will happen on the backend.
What we've learned so far: Even within our work so far, it's becoming clear that what we're actually building isn't so much an AI challenge as a 'one model per asset' challenge. In initial attempts, we found it challenging to get a single model to successfully describe the dynamics of very different types of infrastructure. We are also finding out that accuracy of a model in a test environment is a very different prospect from trusting that model enough to give it control over physical devices - a significant and important gap that informs many of our decisions as we move forward.
