ISTARI

Inspiration

Infrastructure failures can be extremely expensive and dangerous, especially when damage develops internally before there are obvious visual signs. Traditional inspection methods are often manual, time-consuming, and reactive, which means problems may not be discovered until they have already become serious.

We wanted to explore whether multiple sensing methods could be combined to detect structural damage earlier and help identify which infrastructure needs attention first.

What it does

ISTARI is a multimodal infrastructure monitoring system that combines:

Acoustic sensing
Vibration analysis
Computer vision

Instead of relying on a single type of measurement, ISTARI combines information from multiple sensing modalities to estimate the condition of a structure and produce a more useful, actionable health assessment.

Our goal is to detect signs of damage before they become visually obvious or develop into larger and more expensive failures.

How we built it

For our prototype, we collected real-world sensor data using hardware including a Zoom H3-VR ambisonic recorder and a drone/camera system.

We built separate processing pipelines for each sensing modality. Acoustic and vibration signals are analyzed using signal-processing techniques to identify changes that may indicate structural abnormalities, while computer vision is used to inspect visible features such as cracks, deformation, and surface damage.

The results from these pipelines are then combined into a unified assessment that can help identify potentially damaged infrastructure and prioritize further inspection.

Rather than relying entirely on simulated data, we focused on building a real-world proof of concept using measurements collected from actual infrastructure.

Challenges we ran into

One of our biggest challenges was integrating several very different types of sensor data into a single system. Images, acoustic recordings, and vibration measurements all have different sampling rates, noise characteristics, and useful features.

We also had to deal with noisy real-world measurements and determine which features were actually meaningful indicators of structural condition.

Another challenge was building a useful demonstration within the limited time of a hackathon. We had to balance making the system technically meaningful with making sure the full pipeline could actually be demonstrated using real hardware and collected data.

Accomplishments that we're proud of

We are especially proud that we were able to move beyond a purely conceptual or simulated solution and build a prototype around real sensing hardware and real-world data.

We also successfully brought together three different sensing approaches into a single system. Instead of treating acoustic sensing, vibration analysis, and computer vision as separate tools, we designed ISTARI around the idea that they can complement one another and provide a more complete picture of structural health.

Most importantly, we built a working proof of concept that demonstrates how multimodal sensing could be used to make infrastructure inspection more proactive rather than purely reactive.

What we learned

One of the biggest things we learned was that infrastructure monitoring is not just a computer vision problem. A camera can identify visible damage, but many important structural changes can occur before anything is visible on the surface.

This motivated us to combine visual information with acoustic and vibration measurements.

We also learned how important preprocessing, filtering, feature extraction, and sensor alignment are when working with real-world data. Even relatively small differences in measurement conditions can significantly affect the signals being analyzed.

More broadly, the project showed us the value of sensor fusion: different sensing methods have different strengths and weaknesses, and combining them can provide information that none of them could provide as effectively on their own.

What's next for ISTARI

The next step for ISTARI would be to collect larger datasets from both healthy and damaged structures so that the system can learn more reliable relationships between sensor measurements and structural health.

Longer term, the platform could incorporate:

Autonomous drone inspections
Distributed acoustic and vibration sensors
Machine-learning-based anomaly detection
Geographic visualization and infrastructure mapping
Continuous structural health monitoring

Our long-term vision is for ISTARI to help utilities and infrastructure operators move from periodic, reactive inspections toward continuous and proactive monitoring, allowing potential failures to be identified earlier and recovery efforts to be prioritized more effectively.