MAS_Visual_Engine

Our engine uses a shared ViT encoder (Siamese-style) to extract embeddings from two images, then a Temporal ViT and U-Net decoder detect, classify, and score visual changes in one pass.

Comment

Inspiration

The MAS_Visual_Engine is designed to help Formula 1 tracks proactively manage and maintain surface quality by doing three key things:

  1. Detect rubber buildup and grip evolution along the racing line.
  2. Track uneven drying patterns after rain for real-time grip balance estimation.
  3. Monitor curb degradation and surface wear from repeated high-speed impacts.

In Formula 1, the track constantly evolves across sessions and even within race stints. Drivers aggressively attack curbs and explore grip limits. As laps progress, subtle but critical surface changes emerge:

Track Evolution What Happens Why It Matters
Uneven drying Moisture evaporates inconsistently Sudden loss of grip → spins
Curb degradation Repeated corner cutting causes cracks or surface wear Tire punctures / floor damage
Rubber buildup Racing line gains grip, marbles form off-line Loss of control when off-line
Debris accumulation Gravel, tire scraps, carbon pieces appear Tire cuts or Safety Car

These risks often go unnoticed until they trigger penalties, damage, or crashes.
MAS_Visual_Engine aims to detect and quantify these changes before they escalate, enabling data-backed, predictive maintenance of track surfaces.

Currently, our tentative training data consists of normal road images, which capture various surface defects such as cracks, potholes, and wear. Since the underlying physics and visual signatures of degradation (cracks, water patches, rubber deposits) are similar, the same model can be extended and fine-tuned for F1 track surfaces, allowing transfer from road-level understanding to high-speed race conditions.

This will extend to detect micro-cracks and early-stage surface degradation, helping track operators identify and address damage before it becomes structurally significant, thus improving safety, cost-efficiency, and sustainability of racing infrastructure.

What MAS_Visual_Engine Does

By comparing two or more images of the same track section over time (e.g., Turn 7, Lap 5 vs Lap 18), it outputs:

Output Interpretation
Pixel-level change map Highlights evolving regions
Change classification {Uneven drying, rubber buildup, curb damage, debris}
Severity score (0–1) Risk progression level

Alignment with SDG 12: Responsible Consumption and Production

The MAS_Visual_Engine supports sustainable track management and responsible resource utilization by enabling early, precise, and automated detection of surface degradation, curb wear, rubber buildup, and debris accumulation. By shifting from reactive track maintenance to a predictive monitoring approach, the system minimizes unnecessary resurfacing, reduces material and energy consumption, and extends the usable life of racing infrastructure. In doing so, it promotes responsible consumption of resources, enhances operational efficiency, and aligns with the principles of sustainable production and maintenance in motorsport environments.

Tentative Dataset*

Mendeley Pothole Dataset

https://data.mendeley.com/datasets/5bwfg4v4cd/2
Contains annotated road surface videos with segmentation masks for potholes and cracks. The model trained on this can be extended to detect surface degradation on F1 tracks.

How It Works

  1. Two or more images of the same corner are captured at different times.
  2. Both are analyzed patch-by-patch using a shared encoder.
  3. Temporal attention detects how patch textures have evolved.
  4. Spatial attention refines understanding using track geometry context.
  5. A decoder reconstructs a change map.
  6. Final heads output a mask, type, and severity.

Model Architecture

Input

Two or more RGB images at timestamps t-n to t:
I_{t-n}, I_t ∈ ℝ^{3×H×W}

Siamese ViT Encoder (Shared Weights)

Each image is tokenized into patches and passed through a shared Vision Transformer (ViT-B/16):
E1 = ViT(I_{t-n}) → [B, 196, 768] E2 = ViT(I_{t-n+1}) → [B, 196, 768]
En = ViT(I_t) → [B, 196, 768]

Temporal Transformer (Understands Surface Change Over Time)

The patch embeddings are stacked across time and attended:
S = stack(E1,... ,EN) → [B, 196, N, 768]
T = TemporalTransformer(S) → [B, 196, 768]

Spatial Transformer (Captures Track Layout & Propagation)

This layer models how changes relate spatially (e.g., crack spreading or rubber widening):
Z = SpatialTransformer(T) → [B, 196, 768]

U-Net Style Decoder + Multi-Task Heads

Head Output Purpose
Mask 1×224×224 Localize change
Class K classes Surface state change
Severity Scalar Progression intensity

Loss Function

L = L_mask + 0.5 * L_class + 0.2 * L_sev

Why It Matters

Domain Before With MAS_Visual_Engine
F1 Reactive surface issue detection Predictive corner-by-corner risk tracking

Vision

  • Live track risk heatmap for teams & stewards
  • Predictive evolution curves (When will curb X become unsafe?)
  • Integration into FIA monitoring for proactive safety management
  • Proving ground for adaptive visual intelligence in dynamic environments

MAS_Visual_Engine isn’t just a visual model — it’s a time-evolving risk intelligence engine for sustainable and safer racing.

Addendum: Cross-Domain Extension to Manufacturing Quality Assurance

While MAS_Visual_Engine is primarily designed for track evolution detection, its underlying architecture — the **Visual Difference Engine (VDE) — naturally extends to manufacturing defect detection.

In manufacturing, precision is everything. A tiny dent, faint scratch, or slight misalignment can mean the difference between a perfect product and a failed one. Current inspection methods struggle with subtle, progressive defects and environment-dependent variations.

Our VDE approach adapts seamlessly:

  • It models a product’s “normal” appearance across assembly stages using time-series embeddings.
  • New frames are compared to learned baselines via feature-level differencing.
  • Abnormal pixel clusters indicate defects like microcracks, contamination, or early corrosion.
  • The engine is product-agnostic and adapts without full retraining.

CarDD Dataset

https://cardd-ustc.github.io/ Includes high-resolution car images with labeled damage regions such as dents and scratches. Useful for extending visual inspection to F1 cars for automated damage detection.

Kaputt Dataset

https://www.kaputt-dataset.com/ A large-scale defect detection dataset from manufacturing and logistics, focused on identifying visual anomalies across varied conditions. Provides a foundation for general visual difference detection in F1 manufacturing and quality control.

Result: A general-purpose time-evolution detection system that originated for racetracks but scales to factories — proving that if it can survive the demands of Formula 1, it can perform anywhere.

Built With

Share this project:

Updates

Submission history