Traditional surveillance systems can detect objects, but they often lack the context needed to understand what is happening over time. A camera might detect a person and a laptop, but that alone cannot explain what happened before, how an incident developed, or why an alert was triggered.

We wanted to go beyond object detection and build a system that gives the physical world memory, context, and reasoning.

That idea led to SENSE — an Environment Intelligence Platform that transforms camera observations into structured event memory, connects related events into incidents, and helps users investigate what happened through natural-language questions.

SENSE turns raw visual observations into meaningful, searchable environmental intelligence.

Real-time object detection: Processes webcam frames using a YOLOv5n ONNX model and OpenCV to detect people and supported objects.
Object tracking: Associates detections across frames to maintain temporary tracking identities.
Temporal event memory: Records events such as person entry, person departure, object detection, interaction start and end, and unattended-object detection.
Incident correlation: Connects related events into incidents and organizes their roles as triggers, primary events, context, and evidence.
Real-time alerts: Delivers event updates to the operations interface using WebSockets.
AI-powered investigation: Integrates NVIDIA Nemotron to answer natural-language questions using relevant event and incident records retrieved from PostgreSQL.

For example, an operator can ask, “Why was the unattended equipment alert triggered in Workspace A?” SENSE retrieves the relevant incident history and supplies a chronological evidence context to Nemotron to generate an explanation.

Instead of manually reviewing hours of footage, users can investigate recorded events through a structured timeline and an AI-assisted interface.

We built SENSE as a connected pipeline that transforms visual observations into explainable event intelligence.

Computer vision

The frontend captures webcam frames and sends them to a FastAPI backend. OpenCV runs a YOLOv5n ONNX object-detection model, followed by confidence filtering and non-maximum suppression to refine the detections.

Object tracking and temporal reasoning

A centroid-based tracking component associates detections across frames. The event engine evaluates tracked observations, zone membership, proximity, and elapsed time to identify state changes. These observations can then generate structured events instead of remaining isolated frame-level detections.

PostgreSQL event memory

We use PostgreSQL to persist event records and maintain the history needed for investigations. Related events can be associated with incidents, preserving a structured representation of what was observed and how the incident developed.

NVIDIA Nemotron integration

We integrated nvidia/nemotron-3.5-lightning-30b-a3b through NVIDIA's hosted, OpenAI-compatible API. A FastAPI query router classifies questions and directs event-related requests toward the appropriate retrieval workflow.

For incident explanations, the backend retrieves relevant incident and event records, builds a chronological evidence context, and passes that context to Nemotron. This combines database-backed retrieval with language-model reasoning rather than relying on an unconstrained prompt alone.

Operations dashboard

We built the web interface using Next.js, React, and TypeScript. The dashboard brings together camera observations, detection overlays, event history, incidents, alerts, and AI-assisted investigations. HTTP APIs support data exchange, while WebSockets enable real-time updates.

We also used AI-assisted development tools, including v0 and Google Antigravity, to support the development workflow.

Turning detections into meaningful events

Detecting an object in one frame does not explain its significance. We needed to associate detections across frames and evaluate temporal changes to create more useful events.

Maintaining context across time

An investigation may depend on several events occurring at different times. Designing the event memory and incident-correlation workflow required us to preserve relationships between observations rather than treating every detection independently.

Grounding AI-generated explanations

A language model should not invent details that are absent from the recorded evidence. We addressed this by retrieving relevant PostgreSQL records and constructing a chronological evidence context before generating incident explanations. We also recognized that proximity alone cannot establish identity, ownership, or physical contact.

Integrating multiple components

Connecting browser-based camera capture, computer vision, tracking, event processing, database persistence, real-time updates, and AI reasoning into one application required careful coordination between the frontend and backend.

Built an end-to-end platform that connects computer vision, temporal event processing, persistent event memory, and AI-assisted investigation.
Implemented a tracking and event-generation workflow that moves beyond isolated object detections.
Designed a structured incident model that links related events and preserves their evidentiary roles.
Integrated NVIDIA Nemotron into a natural-language investigation workflow backed by retrieved event records.
Created a web-based operations dashboard for viewing visual detections, event history, incidents, and alerts in one place.
Developed a foundation that can be extended toward multi-camera intelligence, edge deployment, and more advanced event correlation.

Our proudest achievement is bringing these components together into a single system built around a simple idea: surveillance should help people understand events, not just watch them happen.

We learned that building a useful AI application requires much more than connecting a model to a user interface.

Computer vision provides observations, but temporal logic is needed to interpret changes. Structured databases preserve the history, while retrieval and language-model reasoning help users investigate that history in natural language.

We also learned the importance of distinguishing observed facts from inferred relationships, designing clear boundaries between deterministic application logic and AI-generated explanations, and building reliable communication between independently developed components.

Working with NVIDIA Nemotron helped us explore how language models can make structured event data more accessible to users who do not want to write database queries manually.

We want to evolve SENSE into a more comprehensive environment-intelligence platform.

Our roadmap includes:

Multi-camera tracking: Associate observations across multiple cameras using more advanced re-identification and tracking methods.
Edge inference: Explore optimized inference on NVIDIA Jetson devices and other suitable edge hardware.
Advanced tracking: Evaluate algorithms such as ByteTrack and BoT-SORT for improved tracking continuity.
Audio-event intelligence: Incorporate supported audio signals, such as glass breaking or industrial alarms, to enrich incident context.
Scalable event processing: Explore Kafka or NATS for handling event streams across larger deployments.
Stronger security and auditing: Add more granular role-based access control and tamper-evident audit records.
Improved evidence-grounded AI: Strengthen uncertainty handling, evidence attribution, and evaluation of generated explanations.

Our long-term vision is to make physical environments easier to understand by connecting observations over time and making the resulting evidence searchable, explainable, and actionable.

SENSE — Giving the physical world memory and reasoning.