VibeSense

Inspiration

It started with a maintenance engineer at a mid-sized food processing plant who carried a screwdriver in his pocket — not to tighten bolts, but to press its handle against a running motor and rest his cheek against the other end. "If I feel a buzz, I know something's off," he told me, completely matter-of-fact.

That image stuck with me. A man with decades of experience, using a 19th-century bone-conduction trick as his primary diagnostic tool — because the modern alternative, industrial vibration sensors, cost more than his entire annual maintenance budget.

The predictive maintenance market is worth \$12.3 billion and growing at 21% CAGR, yet the vast majority of the world's factories still run on gut feeling and screwdrivers pressed to motors. I asked myself: what if the camera already mounted on the wall could do what that screwdriver was doing?

What it does

VibeSense turns an ordinary smartphone or CCTV video of a running machine into a full predictive maintenance report — no sensors, no installation, no specialist required.

Engineer films machine  →  Uploads to VibeSense  →  Gets a health report
     (10 seconds)              (2 minutes)              (3 minutes later)

The report includes:

An anomaly score (0–100) and severity badge (Normal / Warning / Critical)
A predicted failure window ("likely failure within 72 hours")
A SHAP-explained root cause in plain English — not just "something's wrong," but what and why
A timeline chart showing anomaly score evolving second-by-second through the video
A downloadable PDF report an engineer can attach to a work order

Every account manages a fleet of registered machines, receives email alerts on critical detections, and builds a trend history per machine over time.

How we built it

The core insight

A machine about to fail moves differently than a healthy one. Optical flow — a computer vision technique that tracks pixel-level motion between consecutive frames — turns out to be a non-contact vibration sensor.

Given the flow field $\mathbf{F}(x, y, t)$ between frames, the magnitude at each pixel is:

$$M(x, y, t) = \sqrt{F_x^2 + F_y^2}$$

and the frame-averaged magnitude over time, $\bar{M}(t)$, becomes a time series we can analyze exactly like a vibration signal — including running an FFT to find dominant frequencies tied to bearing faults, imbalance harmonics, and resonance.

The pipeline

Next.js 14 (App Router)
      ↕  REST + WebSocket
FastAPI (async)
      ↕  Task dispatch
Celery Workers  ←→  Redis (broker + pub/sub)
      ↕  ML inference
OpenCV → Feature Extractor → Isolation Forest → SHAP
      ↕  Storage
PostgreSQL 16   MinIO (S3-compatible)

Ingest — video uploaded, validated, stored in MinIO; Celery task queued
Extract — Farneback dense optical flow on every frame pair at 320px width
Engineer features — 11 features per 1-second window (magnitude stats, dominant frequency, spectral entropy, spectral centroid, directional circular std, RMS energy, etc.), normalized with RobustScaler
Score — an Isolation Forest scores each window; anomaly score $s(x,n) = 2^{-E[h(x)]/c(n)}$ measures how easily a point is isolated from the "normal" distribution
Explain — SHAP TreeExplainer decomposes the score into per-feature contributions satisfying $\sum_i \phi_i = f(x) - \mathbb{E}[f(X)]$, so every point of deviation is fully accounted for by a ranked, human-readable feature

Progress streams live to the frontend over WebSocket (with polling fallback), so the engineer watches each pipeline step happen in real time.

Challenges we ran into

Cold start with zero labelled data. There's no dataset of "healthy vs. failing" videos for your specific machine. Isolation Forest solves this by requiring only normal data — the first several inspections establish a per-machine baseline, and accuracy improves the longer a customer uses the platform.

Async/sync architecture collision. FastAPI runs on asyncpg; Celery workers are synchronous. Mixing them threw RuntimeError: no running event loop. The fix was two independent SQLAlchemy engines — one async for the API, one sync (psycopg2) for workers — pointed at the same database.

Real-world video quality. Low-light footage produced near-zero flow magnitude, and the model confidently returned "normal" on video where it genuinely couldn't see the machine — a silent false negative. We added a pre-analysis quality check that flags insufficient motion or poor lighting before scoring, so the system never pretends to know something it can't see.

Earning trust from skeptical engineers. Maintenance teams have been burned by black-box "AI monitoring" that cries wolf. We learned the barrier to adoption wasn't raw accuracy — it was interpretability. The SHAP waterfall and plain-English summary aren't UX polish; they're the trust mechanism that lets a 20-year veteran engineer apply their own judgment instead of blindly trusting or dismissing the tool.

Accomplishments that we're proud of

Built a complete, production-shaped system end-to-end in the hackathon window: auth, multi-tenancy, async video pipeline, ML inference, explainability, real-time UI, and email alerting
Designed an ML approach that works with zero labelled training data at launch — a real constraint most predictive-maintenance demos hand-wave away
Made every anomaly score explainable down to the individual feature, satisfying a mathematical completeness guarantee (SHAP's efficiency property), not just a black-box confidence number
Kept the entire architecture swappable — MinIO → S3, Isolation Forest → TCN — without touching the layers above, so the MVP is a real foundation, not a throwaway prototype

What we learned

Optical flow is a legitimate vibration proxy. Spectral entropy and circular directional variance — not raw magnitude — turned out to be the most predictive features, since they capture how organized the motion is, not just how much of it there is.
Unsupervised learning isn't a compromise — it's the correct architecture for a cold-start problem where every customer's "normal" is different.
Explainability is the product, not a feature. For a domain expert to trust an AI verdict, they need to see the reasoning, not just the conclusion.
The hardest constraint wasn't the model — it was the video. Real factory footage is messier than any benchmark dataset, and building for that reality mattered more than squeezing out marginal accuracy gains on clean data.

What's next for VibeSense

RTSP live streaming — always-on analysis from an existing IP camera instead of periodic manual uploads
Supervised TCN model — once user feedback accumulates labelled data, fine-tune a Temporal Convolutional Network (pre-trained on the public MIMII dataset) to push precision from ~72% toward 88%+
Thermal camera support — catch electrical faults (winding insulation, connection resistance) that never show up as motion
A marketplace of machine-specific models — equipment OEMs contribute pre-trained models for their own products, turning VibeSense from a SaaS tool into a platform