Southern Cuts

• 

  Demo Bridge

• 

  schematic

Inspiration

One of the team members, Elliott, comes from a city in California that is prone to natural disasters that worsen structural integrity in bridges, buildings, power grids. These structural failures often become catastrophic because of the common use of reactive maintenance, where maintenance is only done once parts fail. Predictive maintenance, on the other hand, allows for failures to be predictive and repaired before complete failure occurs. This type of maintenance is 3-5x cheaper in the long wrong and presents a 30-50% reduction in downtime.

Edge computing has its advantages over common cloud computing as well. Rather than relying on sensors that need constant internet connection as well as increased latency, a microcontroller equipped with a tinyML model can predict structural failures ahead of time.

What it does

An IMU (accelerometer) is fixed to a structure, reading raw x,y,z movements. The data is then processed using FFT and these frequencies are used as classifiers for a machine learning algorithm. The machine learning algorithm then determines whether this is normal movement, or if the structure begins to fail. All data collection, processing, and machine learning is done entirely locally on an ESP-32 S3 microcontroller.

How we built it

Connected to the ESP-32 is an ABX00101 IMU that is fixed on top of our 3D printed demo bridge. This IMU collects the raw x,y,z data as the bridge moves due to the TT motor we have attached that is spinning with a weight on it. As the motor spins, it sways the bridge in different modes: bold Normal Movement: 3.3V to motor with no weight bold Warning Movement: 3.3V to motor with weight bold Catastrophic/Failing Movement: 5V to motor with weight

As the raw data comes into the board, a Hann Window is applied to clean the data and pre-process it for the next step. This pre-processed data is then run through a Fast Fourier Transform which then outputs low, mid, high frequencies for the collected data. These three frequency types are then used as our three classifiers. The FFT outputs 7 columns: label, infra (sum of FFT magnitudes in bins 1-4), low (sum of FFT magnitudes in bins 5-24), high (sum of FFT magnitudes in bins 50-100), peak_bin (which frequency was dominant), peak_mag (how loud that dominant bin was).

The data was trained as described above and inputted into Qualcomm's Edge Impulse. In Edge Impulse we uploaded our training data and labeled them as normal, warning, failing as above. From there the impulse was created from our 6 input features, selecting (infra, low, mid, high) as our raw data, and then choosing a standard classifier for our classification. From there features were generated, and a neural network was setup with a 50 training cycles, 0.0005 learning rate, and the CPU as the training processor. We used a standard input layer (with our 4 features) + dense layer (20 neurons) + dense layer (20 nuerons). The output is our three (normal, warning, failure) outputs. Once the model was retrained it was deployed as an Arduino model.

From there our program was written and live prediction model was being outputted to the Serial.

Results

Unfortunately, due to our demo setup we had poor training data and thus an average model. Our features were all over the place with an average 51.3% accuracy for the confusion matrix. During live testing, the model seems to be very well at detecting the failure, and getting a little bit confused between normal and warning.

In the end however, we have a model running completely locally with a 0ms computing time and 16 bytes of memory being taken up. This is a huge accomplishment in itself as it proves that we are able to deploy and predict structural failures on a single ESP32-S3.

Challenges we ran into

A 3D print of ours (necessity for training the model) got buried in the queue so we lost a lot time not being able to train and do our testing. This led us to creating a cardboard version of the bridge and training the model with that. This training data was not close to our new 3D printed model so that data had to be scrapped and we had to retrain with new data. This set us back a lot of time. We also started sampling data too slow - 2 Hz and with everything already recorded and time pressing us, it was too late to go back.

Accomplishments that we're proud of

Project worked somewhat in the end despite all of the set backs.

What we learned

Time efficiency, tinyML, Edge Impulse, ESP-32 S3 development, IMU use, motor use.

What's next for Southern Cuts

Applying this to a larger, real model and testing with that. We need hundreds of hours of training data to create a much more accurate predictive model. Also, is the accelerometer feasible for an actual large scale bridge or structure? More sensors could be used such as sound detecting (we can use traffic on bridges or weather noise as another classifier).

Built With

• arduino
• cad
• edge-impulse
• esp32
• imu
• tinyml