Inspiration
We were given a challenge: build a general-purpose visual comparison engine. We quickly realized this is a "holy grail" problem in computer vision. We saw two distinct, conflicting needs:
Manufacturing & Compliance Audits need to be real-time. They happen on a fast-moving production line or in a live video feed.
Infrastructure Degradation & Brand Compliance needs to be robust. It involves comparing images from different times, angles, and lighting conditions (e.g., a sunny-day drone shot vs. a cloudy-day one).
The problem is that fast models are "stupid" (fooled by shadows), and robust models are "slow" (computationally massive). A simple diff tool is useless in the real world due to noise, and a heavy AI model can't run in real-time. Our inspiration was to build a single, elegant system that solves both problems. We didn't want to build a simple "diff tool"; we wanted to build a truly Dynamic Visual Auditor that could intelligently decide how to look at a problem.
We have built the solution for general system by considering various different needs and Applications.
What it does
DVA (Dynamic Visual Audit) is an intelligent, two-stage platform that automatically detects, verifies, and describes visual changes in any time-series of images.
It operates in two modes that work together:
Stage 1: The "Real-Time Filter" (The Sieve) - This is a lightweight, high-speed (30+ FPS) pipeline that monitors a live video feed. Its only job is to be fast and filter out 99% of the "noise" (like sensor jitter or minor lighting flicker). When it detects a potential event, it doesn't alert the user. Instead, it "triggers" Stage 2.
Stage 2: The "DVA-Zero" Audit (The Expert) - This is our core innovation. When triggered, this heavy-duty pipeline analyzes the "suspicious" event.
- First, it uses a Transformer-based model (LoFTR) to perfectly align the new image with the baseline, even if the camera angle is completely different.
- Next, it uses a Foundation Model (DINOv2) to compare the semantic features, not the pixels. This allows it to instantly reject false positives like shadows or glare.
- Finally, if the change is confirmed as real, it feeds the "before" and "after" patches to a Generative Vision-Language Model (LLaVA). Instead of outputting a generic label like "Defect", our DVA describes the change in plain English, such as: a hairline crack has appeared on the support beam or the red logo has been replaced with a blue one.
- The final output for the user is not a noisy video feed, but a clean, text-based log of confirmed, described events.
The Next Mode of the This Stage is : The Scout (Strategic & Competitive Analysis)
This mode is our external intelligence agent, perfect for the application use in Formula1. It compares our design to a competitor's design.
Stage 3: Severity Assessment & Action (The Prioritization Engine) : Furthermore, DVA doesn't just describe the change; it assesses its severity. We feed the generated description (e.g., 'Hairline crack detected') into a lightweight LLM (Gemini Flash) which rates it on a 1-5 scale. This score is displayed in the UI and used to trigger simulated alerts (e.g., Critical alerts for scores 4-5)
How we built it
Our architecture is a decoupled, scalable system designed for both speed and intelligence.
API & Task Queue : A FastAPI backend ingests image data (live or batch) and uses a Celery task queue with Redis to manage workloads. This lets us scale our "DVA-Zero" workers independently.
Storage : MinIO (S3-compatible) stores the baseline and comparison images, while a PostgreSQL database stores the final JSON results and the generated text descriptions.
The "Stage 1" Filter: We built a simple but effective trigger using OpenCV's cv2.absdiff and basic thresholding. It's blazing fast and runs on the CPU, making it a perfect, low-cost "sieve" to catch potential changes. To reduce the load, its safer to use OpenCV along with heavy models.
The "Stage 2" Pipeline :
Alignment: A pre-trained LoFTR model integrated to handle robust, viewpoint-invariant alignment.
Detection: The DINOv2 (Vision Transformer) encoder is used for detection purpose. By comparing the feature vectors (via cosine similarity) instead of pixels, true noise-invariance can be achieved.
Description: The output is connected with patches to a LLaVA VLM. We give it a single, fixed prompt: "You are a 'Dynamic Visual Audit' system. You will be given two images: 'Image A (Before)' and 'Image B (After)'. Concisely describe the single most important change in Image B." This generative, zero-shot approach is our system's "magic."
The Next Feature in this Stage is specifically built for Formula 1:
Input: Image A (Our-Baseline-F1-Wing) vs. Image B (Competitor-F1-Wing-Spotted-on-Track)
1. Sieve: The filter flags a competitor's car in a live feed.
2. (Expert): The "Expert" pipeline aligns the two different wings and semantically compares them. The VLM (LLaVA) is given a "Scout" prompt.
Output: A strategic insight: "Target car has a 5-element front wing flap; baseline only has 4, likely for higher downforce."
3. (Prioritizer): The LLM assesses the output for Strategic Importance: "Importance: 4/5 (High)."
Stage 3: Severity Assessment & Action (The Prioritization Engine)
This final stage takes the text description generated by Stage 2 and adds actionable intelligence:
Severity Scoring: The description is fed to a separate, efficient LLM with a prompt to rate the finding's severity on a scale (e.g., 1-Minor to 5-Critical).
Prioritization: The resulting score allows the system (and users) to immediately understand the importance of the detected change.
(Simulated) Alerting: Based on the severity score, the system logs a simulated action (e.g., "CRITICAL: Alert sent to maintenance team!" for high scores, "INFO: Logged for routine check" for low scores).
Challenges we ran into
The "General-Purpose" Trap: Our biggest challenge. How can one model find both "infrastructure cracks" and "incorrect logos" for brand compliance? We realized a traditional classifier was the wrong path. It's brittle and can only find what it's trained on. This led us to our breakthrough: don't classify, describe. A generative VLM is truly general-purpose because it can describe anything.
The Speed vs. Accuracy War: Our "DVA-Zero" pipeline is brilliant but slow (seconds per frame). Our "Stage 1" filter is fast but dumb. The solution was the cascaded architecture. By using the fast, "dumb" model to protect the slow, "smart" one from unnecessary work, we built a system that is both real-time and intelligent.
The Noise Problem: A simple SSIM or absdiff was a disaster. A person walking by or a cloud's shadow would trigger thousands of false alerts. We can solve this by using DINOv2 as our "semantic gatekeeper." It understands the concept of a "wall" vs. a "shadow on a wall," allowing it to reject 100% of these noise-based false positives.
Accomplishments that we're proud of
True Zero-Shot" Auditing: We can aim the DVA at a completely new domain it has never seen (e.g., from manufacturing to retail shelves), and it just works. Because it describes change rather than classifying it, it requires zero retraining.
Generative, Actionable Insights: We're not just outputting a red box. Our system provides human-readable, descriptive logs. This is the difference between data ("Change detected at [x,y]") and intelligence ("A blue bottle has replaced the red one on shelf 3.").
Solving the "Noise vs. Signal" Problem: Our DINOv2 semantic filter is incredibly effective. Watching it flawlessly ignore a massive shadow moving across a scene while instantly catching a tiny new crack is something our team is extremely proud of.
Prioritized, Actionable Intelligence: By automatically scoring the severity of detected changes using an LLM, DVA moves beyond raw data to provide prioritized insights, allowing users to focus on the most critical issues first.
What we learned
Supervised Classifiers are Brittle: The "general-purpose" prompt was a trick question. We learned that the only way to build a truly general-purpose vision system is to move from classification (a closed world) to description (an open world).
Foundation Models are Building Blocks: The innovation isn't using a VLM. It's in the architecture. We learned to use specific models for specific jobs: LoFTR for geometry, DINOv2 for semantic features, and LLaVA for high-level reasoning.
A Good "Dumb" Filter is Better than a "Medium" Smart One: The cascade was key. It's better to have a blazing-fast, simple filter handling 99% of the work than one "medium" model that tries to do everything and fails at being both fast and robust.
What's next for DVA(dynamic visual audit)
Edge Deployment: We plan to compress the "Stage 1" filter to run entirely on a low-power edge device (like a Jetson Nano or a smart camera). The camera itself can be the "sieve," only sending "suspicious" frames to our cloud-based "DVA-Zero" pipeline for the final audit.
Temporal Reasoning: Right now, we get a list of text-based events. The next step is to feed this log into an LLM to find patterns over time. The LLM could provide a daily summary: "The crack on support beam 4 has grown by 0.2mm this week, and corrosion on flange 2 is accelerating."
3D/NeRF Auditing: We want to expand beyond 2D. By ingesting multiple images, we can build a 3D "Digital Twin" (using NeRF) of an object or scene. DVA can then audit this 3D model, allowing us to detect and measure changes in volume, such as the true size of corrosion or the wear-and-tear on a part from all angles.
Log in or sign up for Devpost to join the conversation.