BlindSpot

Inspiration

Security alerts usually tell you which server looks suspicious, but not why, and definitely not what's physically happening at the rack. Logs, sensor readings, network flows, and the actual hardware live in separate worlds, so responders are left stitching context together by hand. We wanted to close that digital-to-physical gap: an investigator that reasons over real evidence, asks a human before acting, and can send eyes on the ground to the exact machine in question.

What it does

BlindSpot is a cyber-physical security and autonomous inspection system with a command center we call Phantom Command. Its workflow:

Observe: devices report health, services, network flows, and config facts over a signed MQTT channel.
Validate: a gateway checks device identity, schema, sequence freshness, and an HMAC-SHA256 signature, rejecting malformed or replayed packets.
Correlate: stateful backend rules in SpacetimeDB turn meaningful anomalies into incidents, without re-investigating routine healthy metrics.
Investigate: an AI investigator reads stored observations through scoped tools and can propose a targeted inspection.
Authorize: a human Commander approves the mission. The AI suggests; it never dispatches on its own.
Inspect: a rover receives the mission and returns structured inspection evidence.
Reassess: evidence is attached to the incident with hashes and storage references, triggering a fresh investigation cycle.
Visualize: Phantom Command shows incidents, rover status, and a 3D digital twin built from 2D occupancy-map data, with every feed labeled DEMO, REPLAY, or LIVE.

What we verified: our integrated pipeline ran end-to-end on real local services (Mosquitto, the gateway, SpacetimeDB, the Python API, and the mission bridge) using signed simulated telemetry, a simulated rover, and a scripted AI provider. In two fresh runs, a suspicious network flow from a virtual Pi became an incident, an evidence-backed inspection request, a human-approved simulated mission, attached evidence, and a reassessment. The digital twin rendered the occupancy feed in a real browser.

What's in progress: we designed the system around two Raspberry Pi server stations and a camera- and LiDAR-equipped PiCar-X rover. Physical integration and live NVIDIA Nemotron inference are being completed and validated separately.

How we built it

Edge and trust boundary: a Python datacenter simulator models healthy metrics, unexpected services, suspicious flows, offline nodes, and config changes. Devices publish to per-device topics on a Mosquitto broker with identity-specific ACLs, and a TypeScript/Node.js gateway verifies signatures over canonical JSON plus per-device monotonic sequence numbers.
Stateful backend: SpacetimeDB holds devices, observations, incidents, missions, evidence, and approvals. A FastAPI service exposes investigation operations with authorization, audit, and policy layers.
AI investigation: the investigator is wired for NVIDIA Nemotron through NVIDIA's hosted endpoint, and every assessment has to be tied back to stored observations.
Rover seam: an authenticated inspect_cabinet mission contract connects to the PiCar-X autonomy stack. The hardware adapter fails closed when it isn't configured.
Phantom Command: React, TypeScript, Vite, and Three.js (with @react-three/fiber and drei), plus ECharts, Zustand, Radix UI, and Tailwind. It has six workspaces, from the Incident Center to Rover Operations and the Digital Twin.
Testing: pytest, Vitest, Playwright, and type checks across every package, with Docker Compose for local services.

Challenges we ran into

Most of our bugs lived between services, not inside them:

Cross-language signatures: Python serialized 95.0 while JavaScript canonicalized it as 95, so valid signed observations failed HMAC checks. We aligned canonicalization across both sides.
Time units: a mission expiry was sent in milliseconds where SpacetimeDB expected microseconds, so missions were silently rejected.
Enum casing: the database reported states like Completed in a different casing than the bridge expected, so mission lifecycles went untracked until we normalized them.
Evidence that vanished: nested artifact folders weren't being discovered, so successful inspections never attached to incidents.
Restarts: an in-memory permission cache was lost on restart and blocked evidence attachment. The bridge now ensures the persisted grant exists first.

Accomplishments that we're proud of

A merged cross-service integration connecting the telemetry pipeline, scoped investigator, rover evidence bridge, hardware adapter, and 3D twin feed.
Two full incident-to-reassessment runs with targets chosen from evidence, not hardcoded.
Replay-resistant signed telemetry and end-to-end evidence traceability.
A clear line between AI proposals and human authority, with no unrestricted control over robot movement or network defenses.
A browser-verified 3D digital twin with coordinate-frame and provenance indicators.

What we learned

Security tools need evidence provenance and explicit uncertainty, not just alarming dashboards.
Integration breaks at the contracts: number formats, clocks, enum casing, identity enrollment, and stale data.
LLMs belong behind bounded tools and approved actions. Model reasoning, mission authorization, and motor safety are separate trust domains.
A 2D occupancy grid shown in 3D helps people orient, but extruded walls aren't measured heights, and we label them that way.
Simulation-first testing let us validate the whole flow while the hardware was still being prepared.

What's next for BlindSpot

Run a live NVIDIA Nemotron acceptance test with evidence-grounded tool calls.
Enroll both physical Pis with their own MQTT and HMAC identities, and ingest real signed telemetry.
Connect the PiCar-X mission API, marker recognition, and camera evidence.
Stream real LiDAR SLAM maps and rover pose into the 3D twin.
Repeat the entire physical workflow on one shared network, then explore policy-gated containment, always behind human approval.

GitHub repository