Inspiration
Inspiration
Vision systems rarely fail loudly. A smudged lens, a bumped mount or a dark evening silently degrades every model downstream, and nothing raises an alert. Most work improves the model; NETRA repairs the camera feed instead, and tells maintenance what is wrong before it gets worse.
What it does
Given a reference frame from the healthy camera, NETRA diagnoses six faults (blur, low light, overexposure, noise, camera drift, lens dirt), picks a repair from a fixed OpenCV tool registry, applies it, re-diagnoses to verify it helped, rolls it back if not, and escalates what it cannot fix. Medium-risk changes (re-alignment, regional edits) need human approval; with no approver they are denied.
Beyond a single frame:
- Health Passport: records every diagnosis per camera and forecasts when a fault limit will be crossed (Theil-Sen trend plus Mann-Kendall test), so maintenance can happen before failure.
- Fleet queue: ranks many cameras into one maintenance list by urgency (simulated fleet, real ranking logic).
- Downstream-task guard: rolls back any repair that lowers the user's own task score, even when image-quality metrics improve.
- Provenance: a hash-chained, tamper-evident audit log and a repair receipt, so a repaired frame cannot pass as the original. This is integrity evidence, not a digital signature.
- Shareable HTML health report per repair, and a live webcam mode for a manual real-camera demo.
How we built it
Python and OpenCV 5.0: ORB + RANSAC and ECC alignment, Laplacian and Immerkaer measurements, CLAHE, bilateral and non-local-means filters, cv2.dft Wiener deconvolution, and inpainting. A rule-based planner is the default; an optional LLM planner can only propose actions that are validated against the diagnosis and the tool registry (tested with mock models only). Streamlit UI, CLI, 28 automated tests. Runs on CPU, about 650 ms per frame.
Results on fresh held-out scenes (192 faulty + 16 clean frames):
- Mean SSIM 0.498 (faulty) to 0.843 (NETRA); PSNR 14.2 to 22.9 dB.
- 0% of frames made worse; a naive "always enhance" pipeline made 71% worse.
- 0 of 16 false alarms on clean frames.
- Task guard (QR codes decoded in fixed regions): 0.72 to 0.79, and frames where the task got worse than the input: 17 to 0, at the cost of SSIM 0.870 to 0.829.
- Health Passport on simulated slow degradations, forecast issued on day 30: drift 6/6 (1.1 days mean error), noise 6/6 (1.1), focus loss 6/6 (2.3), dimming 6/6 (4.9); 0 false alerts on 24 checks of stable cameras. Early lens-dirt forecasts are poor (about 16 days error).
Challenges we ran into
Our own evaluation exposed a flaw: after exposure repairs, clipped regions looked like lens smudges (73 false alerts). A first fix made it worse and was discarded; excluding pixels the sensor clipped fixed it (calibration 5/20 to 0/20 false alarms). We then re-ran on fresh scenes and report only those.
Linear forecasting also failed for focus loss at first (0/6). Looking at several time windows fixed it, and we kept the earlier results in the repo for comparison.
Blur remained the hardest fault: it is diagnosed correctly but mostly not repairable.
Accomplishments that we're proud of
An agent that checks its own work, refuses to act without permission, and reports its failures honestly, including what it cannot fix. It never made a frame worse on held-out scenes, and it raised no false alarms on clean frames.
What we learned
- A repair is only useful if it is verified. Re-diagnosing after every fix, and rolling back when it does not help, is what separates NETRA from a naive "always enhance" pipeline.
- Image-quality metrics are not the same as task performance, which is why the downstream-task guard exists.
- Evaluating on scenes we never tuned on exposed problems that tuning-set results hid, and fixing them honestly mattered more than a good headline number.
- Forecasting needs the right window: a single linear trend failed for some faults until we looked at several time windows.
- Human approval is a design feature. Letting the agent act only on low-risk changes made it easier to trust.
What's next for NETRA
Long-term real camera footage for the Passport, learned deblurring behind the same verifier, signed audit heads, and cloud deployment.
Current limitations: faults and the fleet are simulated and there is no validated real-camera evaluation; there is no AWS deployment; the LLM planner was tested with mocks only; and everything is judged against a reference frame from the healthy camera.
Log in or sign up for Devpost to join the conversation.