CamBrain

detection of forbidden object's

Inspiration

Cameras are everywhere, but they're dumb recorders. Someone has to sit and watch a wall of screens, and incidents are only discovered after the fact — a fall in a nursing home, a worker skipping PPE, a phone sneaked into an exam hall. The "smart CCTV" vendors that exist today fix this by uploading footage to the cloud, which is a hard no in the environments that need video AI most: hospitals, schools, and factories governed by GDPR, India's DPDP Act, HIPAA, and FERPA. On top of that, they ship static models that never learn a specific site.

We wanted the opposite: video intelligence that runs entirely on a single consumer GPU, on-premise, with a zero-disk-write privacy mode — and an engine that fuses detection, behavior analytics, and a vision-language model into one runtime-switchable system.

What it does

CamBrain turns any camera into an on-device, privacy-first intelligence sensor with four runtime-switchable operational modes:

General — fall detection (pose-LSTM + rule fusion), face recognition, behavioral-IQ baselines
ElderCare — medicine-reach detection, inactivity alerts, night bed↔bathroom tracking
Exam — gaze tracking, forbidden-object detection, desk-zone breach, collusion heuristics
Safety — PPE compliance, danger zones, time-to-collision warnings

An intelligence layer (Groq Llama-4-Scout VLM, ~0.44 s) auto-narrates every critical event, and /api/chat answers natural-language questions over the live view and event history — "did anyone enter the server room after 6pm?". Cross-camera Re-ID (OSNet embeddings + HSV fallback) keeps a persistent identity across the whole site. All of it runs live at cambrain-web.vercel.app.

How we built it

Stack: FastAPI + WebSocket · Ultralytics YOLO11 · ByteTrack · MediaPipe · DeepFace/InsightFace · Groq → Gemini → Ollama VLM cascade · SQLite/Postgres · Next.js 16 marketing site.

Custom-trained models (our own weights, not off-the-shelf):

ppe.pt — construction-PPE detector, yolo11s, 100 epochs → mAP50 0.845, real-world verified on a YouTube worksite clip (Person .95, Hardhat .94, Vest .92)
forbidden.pt — exam-cheat detector, yolo11s, 120 epochs → mAP50 0.805, 6/6 class match on real exam imagery
fall_lstm.pt — pose-sequence LSTM with velocity features: a 38-dim stateful feature vector (34 pose features + torso angle + aspect + 2 frame-to-frame velocity terms $\Delta c_y,\; \Delta \text{aspect}$) that catches the falling transition, not just the lying result:

$$ \mathbf{v}t = \big[\, \Delta c_y,\; \Delta \text{aspect} \,\big], \qquad \Delta x_t = x_t - x{t-1} $$

Recall 0.90 on labeled data; velocity drives max fall-probability 1.000 on all 3 real fall clips.

Architecture decisions that mattered: detection stays rule-based and local; the VLM only explains what detectors already flagged (single-frame VLM vision judgment proved unreliable as a detector). A motion-gated sentry-sleep (1 FPS when idle) plus strided MediaPipe gives ~12 cameras on one RTX 4050. The self-learning loop (VLM teacher → pseudo-label → fine-tune → promotion gate) is our moat. Hardened for production: API-key auth, CORS lock, face/plate redaction, and 4/4 repo-hygiene checks — no secrets or weights ever tracked.

Challenges we ran into

A silent training disaster: our PPE-v2 retrain produced a degenerate model that detected only Hardhat/Vest. Root cause: a case-sensitive split-folder lookup on Kaggle's Linux filesystem silently matched zero images for the primary dataset. Fixed with case-insensitive split detection and a pre-flight gate that aborts in seconds if any class has <30 boxes — no more burning GPU hours on a doomed run.
Identity churn: HSV color-histogram Re-ID collapsed or split identities (a person on the floor has a "lying torso" that no longer matches their standing appearance). Swapping in learned OSNet embeddings cut incorrect merges from 49% → 9%.
Fall-detector false positives: single-frame LSTM scoring produced fall≈normal probability saturation. Solved with temporal smoothing — a fall is confirmed only when a strict majority of the last 5 frames exceed the threshold:

$$ \text{confirmed\fall}(t) \iff \sum{k=0}^{4} \mathbf{1}\big[\, p_{\text{fall}}(t-k) > \theta \,\big] \ge 4 $$

…plus a corroboration-only policy: the LSTM can escalate a rule-based BENDING to FALL but never fires standalone.

Performance on one GPU: MediaPipe FaceLandmarker crushed Exam-mode fps. Striding per-track ($\text{FACE\_LM\_STRIDE}=3$) recovered ~2.7 → ~8.3 fps/cam.
Privacy-first by construction: publishing a self-contained public repo while keeping the full engine private — scrubbing VPS credentials, keeping model weights and test data out of git, and making the public hackathon repo fully cloneable.

Accomplishments that we're proud of

Real-time on one consumer GPU: base detection 26 ms/frame (38 fps), ~4 cameras @ 5 fps, ~12 with motion-gating.
Four domains, one engine, switched at runtime — no single-vertical lock-in.
Custom-trained heads with measured, published metrics — including the velocity-feature fall LSTM (recall 0.90).
A working privacy proof: a zero-disk-write dashboard with real-time RAM traces that prove frames never touch disk.
Conversational intelligence: deterministic factual recall over the event log plus VLM-powered Q&A — no hallucinated safety verdicts.
Production-grade engineering: CI green, Dockerized, stress-tested across all 4 modes, repo-hygiene green, cross-camera persistent identities.

What we learned

Privacy is the moat. Regulated organizations literally cannot use cloud-upload CCTV — an on-prem, zero-disk-write engine unlocks a market the incumbents can't serve.
Detection is commoditized; per-site learning is not. The differentiation is a VLM teacher that self-labels each deployment's blind spots, with a promotion gate that rejects regressed models.
Architecture honesty pays. VLM for narration/triage, rule-based detectors for safety — knowing what a model isn't good at (single-frame vision judgment) prevented a dangerous design.
Constraints breed efficiency. Building for one GPU forced motion-gating, inference stride, sentry-sleep, and ONNX paths — exactly the levers that make it deployable on hardware organizations already own.

What's next for CamBrain

Deploy the continuous self-learning loop in real sites — per-camera blind-spot adaptation via VLM-as-teacher labeling
More camera sources: battery/intermittent-stream proxies (Eufy/Reolink), cloud adapters, and Milestone/Genetec VMS plug-ins for enterprise
Deep privacy: LoRA-only deltas, federated averaging, and differential privacy for the training loop
3× faster inference via TensorRT → more cameras per box; a packaged app installer with bundled offline VLM (Ollama + qwen2-vl) for zero-setup air-gapped tiers
Real-world pilots in elder care, hospital safety, and exam integrity — proving privacy-native video intelligence where it's needed most