TRINETRA

An AI-powered 3D UAV training simulator that combines RGB, thermal, and radar data to recreate drone-threat scenarios, evaluate trainee decisions, and adapt future training based on performance.

Comment

TRINETRA

  1. Project Links
  1. Project Story

The Idea Behind TRINETRA

In 2026, small UAVs are becoming increasingly accessible, capable and difficult to monitor across complex environments. Preparing operators for these situations requires training environments that can reproduce different threat scenarios, sensor limitations and environmental conditions.

We started building TRINETRA around a simple question:

How can we create a training environment where drone-threat scenarios can be simulated repeatedly, multiple sensors can be evaluated together, and human decisions can be measured objectively?

TRINETRA is our attempt to answer that question through a combination of 3D simulation, multimodal AI perception, probabilistic sensor fusion and adaptive training.

The platform creates configurable UAV scenarios, processes RGB, thermal and radar observations, maintains a shared representation of the simulated environment and evaluates trainee performance throughout each session.

  1. System Architecture and Technical Implementation

We designed TRINETRA as a connected pipeline consisting of six major stages.

3.1 Scenario Generation and Simulation

The simulation environment is built using NVIDIA Isaac Sim. It supports configurable UAV situations, environmental conditions, target densities and sensor parameters.

Training scenarios can include variations in:

  • Weather and illumination.
  • UAV positions and movement.
  • Target density and background complexity.
  • Sensor availability and degradation.
  • Visibility and observation uncertainty.

This allows the same scenario to be recreated under controlled conditions, making performance comparisons more consistent.

3.2 Multimodal Sensor Processing

TRINETRA processes three complementary observation streams:

RGB: Provides visual information about objects, backgrounds and spatial context.

Thermal: Provides infrared observations that can help identify UAVs under challenging lighting and visibility conditions.

Radar: Adds another source of observation for the simulated environment.

Each modality contributes different information. Their combination provides a broader representation of the environment than any individual observation stream.

3.3 AI-Based Detection and Tracking

The perception pipeline uses AI models to detect and track UAVs across incoming observations.

The system also considers:

  • Detection confidence.
  • Sensor reliability.
  • Duplicate observations.
  • Unfamiliar or out-of-distribution objects.
  • Uncertainty in object localisation and tracking.

The objective is to maintain consistent object information while accounting for changes in observation quality.

3.4 Probabilistic Sensor Fusion

One of the central components of TRINETRA is its probabilistic world-state estimation.

Observations from RGB, thermal and radar are processed and combined into a shared representation of the environment.

The underlying Bayesian estimation principle can be expressed as:

[ P(X_t\mid Z_{1:t}) \propto P(Z_t\mid X_t)P(X_t\mid Z_{1:t-1}) ]

Where:

  • (X_t) represents the estimated environment state.
  • (Z_t) represents observations at time (t).
  • (P(X_t\mid Z_{1:t})) represents the updated state estimate.
  • (P(Z_t\mid X_t)) represents the observation likelihood.

Sensor importance is also adjusted according to observation quality. Conceptually, confidence-based weighting can be represented as:

[ w_i=\frac{q_i}{\sum_{j=1}^{n}q_j} ]

Here, (q_i) represents the quality assigned to sensor (i), and (w_i) represents its relative contribution.

This allows the fusion process to account for degraded observations rather than treating every sensor measurement as equally reliable.

3.5 Trainee Interface and Decision Evaluation

The trainee interacts with the simulated environment through a dedicated interface.

The system records decisions, response times and errors during each session. These events are connected to the observations available at the time of each decision.

This makes it possible to examine:

  • What the trainee observed.
  • Which information was available.
  • How quickly the trainee responded.
  • What decision was made.
  • Where the decision could be improved.

3.6 Adaptive Training and Counterfactual Replay

After a training session, the evaluation engine analyses the trainee's performance and identifies areas requiring improvement.

Future scenarios can then be adjusted to address individual weaknesses.

We also designed counterfactual replay, which allows the same simulation state to be branched so that an actual decision can be compared with an alternative decision and its resulting outcome.

This provides a way to examine decisions under consistent simulated conditions.

  1. What Makes the Approach Different?

Quality-Aware Sensor Fusion

The system adjusts sensor importance according to observation quality, reducing the influence of unreliable measurements.

Open-Set Object Awareness

The perception pipeline considers unfamiliar and out-of-distribution objects rather than assuming that every observation belongs to a known category.

Learned and Probabilistic Estimation

Multimodal temporal features are combined with probabilistic state estimation, bringing learned sensor relationships and interpretable state estimates into the same pipeline.

Simulation-to-Real Validation

The platform supports evaluation of how models developed using simulated data transfer to real sensor observations.

Counterfactual Decision Analysis

Training sessions can be replayed and branched to compare different decisions under the same initial simulation conditions.

Human-AI Performance Evaluation

The system evaluates AI performance and human training performance separately, allowing us to examine whether AI assistance actually improves trainee outcomes.

  1. Challenges We Faced

Building a system that connects simulation, multiple sensing modalities and human decision evaluation introduced several engineering challenges.

Sensor Heterogeneity: RGB, thermal and radar observations have different characteristics. Combining them requires careful consideration of observation quality and uncertainty.

Unreliable Observations: Sensor degradation, environmental changes and false detections can affect the estimated environment state. We incorporated confidence and uncertainty considerations into the perception and fusion pipeline.

Duplicate Evidence: Multiple observations can refer to the same simulated object. Persistent tracking and event association are necessary to avoid treating duplicate evidence as separate targets.

Unfamiliar Objects: A model may encounter observations outside its known categories. We considered out-of-distribution detection as part of the system design.

Human Performance Evaluation: Evaluating detection accuracy alone does not explain whether an operator makes better decisions. We therefore designed separate evaluation paths for AI outputs and trainee performance.

Simulation-to-Real Transfer: Performance in a controlled simulation does not automatically guarantee equivalent performance with real sensor data. This makes transfer evaluation an important part of the platform.

  1. Technology Stack

Component| Technologies Simulation| NVIDIA Isaac Sim AI / ML| PyTorch, TensorFlow, NVIDIA TAO Computer Vision| OpenCV Backend| Python, FastAPI Frontend| TypeScript, Three.js Systems Programming| C++ Robotics Integration| ROS2, Arduino Database| PostgreSQL Deployment| Docker, AWS, Vercel

  1. What We Learned

Working on TRINETRA helped us understand the complexity of building systems where multiple AI components must operate together.

We explored how different sensing modalities contribute to environmental perception, how uncertainty affects state estimation and why reliable tracking requires more than individual object detections.

We also learned that evaluating an AI system requires looking beyond model-level metrics. In a training environment, the quality of human decisions, response time and the information available at the time of a decision are equally relevant.

The project brought together concepts from computer vision, probabilistic estimation, sensor fusion, simulation engineering and human-computer interaction.

  1. Future Scope

TRINETRA is designed to support further expansion through:

  • Additional sensor modalities and perception models.
  • More complex UAV scenarios and environmental conditions.
  • Improved real-time inference on edge hardware.
  • Expanded simulation-to-real validation.
  • More detailed trainee performance analytics.
  • Multi-unit training environments.

Our longer-term direction is to develop a scalable platform for repeatable UAV threat training, with configurable scenarios and measurable human-AI performance.

  1. About the Project

TRINETRA is an AI-powered 3D UAV training simulator that combines RGB, thermal and radar observations to recreate drone-threat scenarios, detect and track UAVs, and evaluate human decisions.

The platform integrates NVIDIA Isaac Sim, multimodal AI perception, probabilistic state estimation and adaptive scenario generation. It maintains a shared representation of the simulated environment while tracking sensor reliability and observation uncertainty.

Trainees interact with simulated situations, make decisions and receive performance feedback. The evaluation engine analyses decisions, response times and errors to help instructors identify weaknesses and configure subsequent training scenarios.

Key capabilities: Multimodal sensor fusion, UAV detection and tracking, uncertainty estimation, open-set object awareness, counterfactual replay, adaptive training and simulation-to-real validation.

Built by Team Star Busters.

Built With

Share this project:

Updates

Submission history