AEGIS-X: AI-Powered Predictive Safety Intelligence

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Inspiration

I was inspired by a simple problem: traditional CCTV systems can show an accident, but they usually cannot help prevent one.

Construction sites and industrial environments contain moving workers, heavy machinery, blind spots, and constantly changing conditions. A human supervisor cannot continuously monitor every interaction and predict which movement could become dangerous.

I wanted to explore whether computer vision could go beyond simply detecting objects and instead understand relationships, movement, and potential risk.

That idea became AEGIS-X — a Predictive Construction Safety Intelligence Assistant.

My goal was to transform a normal camera feed into a system that could detect what is happening, track how entities are moving, estimate potential collision risk, and explain that risk to a safety supervisor.

How I Built It

I built AEGIS-X as an integrated computer-vision and AI system rather than a standalone object detector.

The core pipeline is:

Detection → Tracking → Spatial Analysis → Velocity Estimation → Time-To-Impact → Risk Prediction → AI Explanation

I trained a custom YOLOv8 model to detect relevant safety entities such as personnel, PPE, and machinery. I then integrated ByteTrack to maintain persistent identities across video frames.

Using the tracked bounding boxes, I developed spatial calculations to estimate the proximity and relative movement between entities. These values form the basis of the Time-To-Impact (TTI) engine, which estimates whether approaching entities may enter a dangerous state.

I built the command center using Streamlit, with OpenCV for video processing, NumPy for mathematical calculations, and Plotly for real-time analytics.

I also developed a Counterfactual Environment Simulator that allows safety conditions to be modified and their effect on calculated risk levels to be observed.

For the AI Copilot, I used Ollama with LLaMA-3 locally. Instead of sending raw video to the language model, I convert the important information from the vision system into a structured World State JSON. The Copilot can then interpret this state and provide natural-language explanations of the current safety situation.

I also prototyped face recognition mapping to associate tracked safety violations with known personnel.

What I Learned

Building AEGIS-X taught me that object detection alone is not enough to create predictive intelligence.

A bounding box tells me where an object is, but it does not tell me where that object is going or whether its movement could create a dangerous interaction.

I learned how to combine computer vision, object tracking, spatial mathematics, real-time processing, structured data, and generative AI into a single system.

I also learned that real-time AI is not only about model accuracy. Latency, FPS, tracking stability, memory usage, and system architecture can have a major impact on whether a system is actually usable.

Most importantly, I learned how to turn outputs from multiple AI components into a meaningful operational workflow rather than treating each model as an isolated feature.

Challenges I Faced

Real-Time Performance

One of my biggest challenges was maintaining real-time performance while simultaneously running detection, tracking, risk calculations, dashboard updates, and state generation.

My initial YOLO inference pipeline operated at approximately 16 FPS under the tested configuration. I optimized the pipeline using reduced inference resolution and frame-skipping techniques.

The optimized AEGIS-X Ultra-Fast Engine reached approximately 31 FPS on average, with observed performance ranging from approximately 27–35 FPS under load.

Converting Detection Into Prediction

Another major challenge was turning object detections into meaningful risk predictions.

Pixel distance is not the same as real-world distance, and camera perspective can distort spatial relationships. For the current prototype, I use calibrated pixel-space thresholds and relative movement information as a practical approximation.

This limitation also showed me where the system needs to go next: true 3D spatial calibration, depth estimation, and multi-camera tracking.

Integrating Computer Vision With an LLM

I also had to determine how a language model could understand information coming from a real-time vision system.

Rather than providing raw video to the LLM, I designed a structured World State JSON containing detected entities, relationships, and risk information. This creates a bridge between the computer-vision layer and the AI Copilot.

This approach also allows the LLM to run locally, which is important when dealing with potentially sensitive industrial environments.

Building the System as One Product

Another challenge was integrating everything into a single usable interface.

Instead of creating separate demonstrations for detection, tracking, prediction, simulation, and AI interaction, I connected them through a unified Command Center Dashboard.

This forced me to think beyond individual models and focus on the complete user workflow: What does the system see? What does it predict? Why is it dangerous? And how can a supervisor understand the situation quickly?

What Makes AEGIS-X Different

The central idea behind AEGIS-X is the transition from reactive detection to predictive safety intelligence.

A conventional safety system might answer:

“What is happening right now?”

AEGIS-X is designed to move toward answering:

“What could happen next, why is it risky, and what should the supervisor know?”

By combining detection, tracking, spatial reasoning, Time-To-Impact analysis, environmental simulation, and a local AI Copilot, I built AEGIS-X as a foundation for a more proactive approach to industrial safety.

What's Next

AEGIS-X is currently a working prototype, and there are several areas I want to improve.

My planned next steps include:

• True 3D spatial calibration and depth estimation
• Multi-camera tracking and spatial fusion
• More advanced trajectory prediction
• Improved environmental risk models
• Mobile and wearable safety alerts
• Historical safety analytics and risk trends
• Larger-scale industrial deployment

The long-term vision is simple:

Turn industrial cameras from systems that merely record accidents into intelligent systems that help predict and prevent them.

Built With

• ai-safety
• artificial-intelligence
• bytetrack
• computer-vision
• data-analytics
• deep-learning
• facial-recognition
• generative-ai
• large-language-models
• llama-3
• machine-learning
• numpy
• object-detection
• object-tracking
• ollama
• opencv
• plotly
• ppe-detection
• predictive-analytics
• python
• real-time-ai
• spatial-computing
• streamlit
• yolov8