Project Banner
Project Architecture Flow
Project Cloud Architecture
Agent Investigation Flow
Erlang AI Agent

From Passive CCTV to Intelligent Surveillance Agents: Building Erlang AI Vision with Qwen and Alibaba Cloud

Inspiration

Our project began with a simple question:

Why can modern surveillance cameras record everything, yet still fail to tell users when something important is happening?

Imagine that at 2:17 a.m., someone appears near the rear entrance of a small shop. The camera records the person approaching the door, waiting for several minutes, leaving, and returning again. The footage exists, but unless someone is watching at that exact moment, nothing happens. By the time the owner reviews the recording the next morning, the event is already over.

This is the core limitation of conventional CCTV systems: they remember well, but they understand very little.

Modern cameras can detect motion, people, vehicles, and sounds, but most still rely on rigid configurations such as fixed zones, object labels, schedules, and confidence thresholds. A person walking past a shop and someone lingering beside a restricted entrance may trigger the same alert, even though the situations are very different.

We believed users should not need to think like computer-vision engineers. They should be able to describe what matters in plain language:

“Alert me if someone stays near the rear entrance for more than 20 seconds after closing time.”

That sentence became the foundation of Erlang AI Vision, an agentic surveillance platform that can:

Understand natural-language monitoring rules
Detect activity locally at the edge
Escalate only relevant events for contextual verification
Gather additional evidence when necessary
Explain why an alert was triggered
Keep camera actions and AI decisions auditable

Our goal was not simply to build another smart camera. We wanted to transform a passive recording device into an active surveillance agent that can observe, investigate, and explain events without sending continuous private footage to the cloud.

What We Built

Erlang AI Vision turns an ordinary camera into a configurable AI agent.

Users describe a monitoring rule in natural language. The platform converts the request into structured detection conditions, continuously evaluates activity at the edge, and sends only relevant candidate events to Qwen for contextual verification.

By the end of the hackathon, we had built an end-to-end prototype that could:

Create surveillance rules from natural-language instructions
Capture video using an ESP32-S3 camera
Run local object and audio detection
Perform first-pass event triage with a small local Qwen model
Verify ambiguous events using Qwen in the cloud
Request additional snapshots or camera movement when evidence is incomplete
Deliver verified alerts through a Flutter web and Android application
Let users query events and control approved camera functions through a conversational assistant
Continue local monitoring during network interruptions
Record camera commands and AI tool calls in an audit trail

The design principle that guided the project was:

Detect locally. Verify intelligently. Alert selectively.

How We Built It

The system spans four main layers: the camera, the edge bridge, the cloud backend, and the user application.

1. Camera Layer

We used a XIAO ESP32-S3 camera module mounted on a pan-and-tilt servo assembly.

The camera:

Captures JPEG frames
Streams VGA footage at approximately 15 FPS over USB
Supports lower-resolution streaming at higher frame rates over Wi-Fi
Scans a pairing QR code during provisioning
Executes validated pan-and-tilt commands
Responds to authenticated snapshot requests

The ESP32-S3 is intentionally responsible only for lightweight camera and actuator operations. Heavier AI workloads are handled by the edge bridge.

2. Edge AI Layer

For the prototype, we used an ordinary laptop as the local edge gateway.

The edge bridge runs:

YOLO26-nano for object detection
YAMNet for audio-event detection
Qwen3.5 0.8B through Ollama for first-pass visual triage
Local rule filtering and dwell-time evaluation
Event clip recording
Offline event buffering
A persistent outbound WebSocket connection to the cloud backend

Routine activity is rejected locally and never sent to the cloud. Only candidate events that may match a user’s rule are escalated.

This allows the system to reduce bandwidth, control cloud-model costs, and establish a meaningful privacy boundary around continuous footage.

3. Cloud Backend

The backend is built with FastAPI and deployed on Alibaba Cloud.

Candidate events are submitted to Qwen 3.7 Plus, which evaluates the selected evidence against the user’s actual monitoring rule. The model returns a verdict and a human-readable explanation.

The cloud stack includes:

Alibaba Cloud Elastic Container Instance for the backend
Alibaba Cloud Container Registry for container images
ApsaraDB RDS for PostgreSQL event and configuration data
Object Storage Service for evidence clips and application assets
Alibaba Cloud Model Studio and DashScope for Qwen inference
Firebase Authentication for user identity
Firebase Cloud Messaging for push notifications
Secret-management controls for runtime credentials

The application infrastructure runs primarily in Alibaba Cloud’s Kuala Lumpur region, while Qwen inference is accessed through the Singapore endpoint.

4. Flutter Application

We built a Flutter web and Android application that provides:

Natural-language agent creation
Live camera viewing
Camera status monitoring
On-demand recording
Event history
Evidence-clip playback
Real-time event updates through Server-Sent Events
Push notifications for high-severity events
A conversational surveillance assistant

Evidence clips are encoded as H.264 so that browsers can play them inline rather than forcing users to download the files.

Why Qwen Was Essential

Traditional object detection can identify what is visible, but it cannot always determine why an event matters.

Erlang AI Vision uses Qwen in three major parts of the platform.

Natural-Language Agent Builder

Qwen 3.7 Max converts a plain-language instruction into a structured monitoring configuration, including:

Relevant object classes
Active schedules
Dwell-time requirements
Confidence thresholds
Cooldown periods
Optional regions of interest

We also created a deterministic keyword-based fallback compiler so that basic rule creation can still work when cloud inference is unavailable.

Contextual Event Verification

YOLO identifies what is present. Qwen determines what the situation means.

For example, a delivery driver approaching a door and a person repeatedly returning to a restricted entrance may contain the same detected object class: person. However, the surrounding evidence, timing, behavior, and user-defined rule make the two events very different.

Qwen reviews the event evidence against the monitoring rule and returns:

Whether the event should trigger an alert
Why the evidence does or does not match
Whether additional evidence is required
A concise explanation for the user

Conversational Surveillance Assistant

Qwen 3.7 Max also powers the in-app assistant.

Users can ask questions such as:

“Show me all events detected at the rear entrance last night.”

“Is the rear camera online?”

“Pan the camera right and capture a new snapshot.”

The assistant can retrieve event history, summarize activity, explain alerts, check camera status, and request approved camera operations.

What Makes the System Agentic

Erlang AI Vision does more than classify events.

When the available evidence is unclear, Qwen can request another observation before making a decision. For example, if a person is partly outside the frame, the model can propose that the camera pan slightly, capture another snapshot, and re-evaluate the event.

However, the model never controls the hardware directly.

Every requested action is:

Sent through the backend
Checked against the authenticated user
Validated against an approved tool schema
Rate-limited
Clamped to servo-safe movement ranges
Executed through the edge gateway
Recorded in the audit log

This separation became one of our most important design decisions:

The model can reason about what should happen, but deterministic software controls how it happens.

The conversational assistant uses a Model Context Protocol tool server to interact with cameras, recordings, and events. If the tool server is unavailable, the assistant safely falls back to text-only responses.

Our Build Journey

Our team consisted of four AI and software enthusiasts from Malaysia: two AI engineers, Fang Wei and Nicholas, and two software engineers, Kenneth and Wei Kiat.

The project required us to connect four very different engineering layers:

Embedded camera firmware
Local edge AI
Cloud infrastructure
A cross-platform Flutter application

This made integration the hardest part of the project. A small change in one layer could affect the entire system.

During development, we dealt with:

Servo calibration problems
Unstable WebSocket connections
Device-state synchronization
Video encoding and browser playback
Cloud deployment failures
Database migration issues
Small-model output inconsistencies
Network interruptions
Camera authentication on an untrusted local network

Many of our key design decisions came from repeatedly asking the same questions:

Which decisions should be made by AI?
Which controls must remain deterministic?
Which data should remain local?
What should happen when the network fails?
How can the system verify a camera without exposing cloud credentials?
How can we prevent a model-generated tool request from becoming an unsafe hardware action?

Challenges We Faced

Challenge 1: The Cloud Could Not Call the Camera

The ESP32 camera sits inside a private home or business network. The cloud backend cannot directly reach it because of routers, firewalls, and network address translation.

Our first architectural challenge was therefore not AI—it was connectivity.

We solved this by making the edge bridge maintain a persistent outbound WebSocket connection to the backend. Because the connection starts from inside the local network, it does not require port forwarding or an exposed camera endpoint.

Every command follows the same audited path:

User or Qwen request
        ↓
FastAPI backend validation
        ↓
Persistent outbound WebSocket
        ↓
Edge bridge
        ↓
ESP32 camera or servo

This path is used for pan, tilt, snapshot, and device-status operations.

One command path. Fully audited. No port forwarding. No exposed cameras.

Challenge 2: The Small Model Silently Rejected Everything

Our edge triage model is a small Qwen3.5 0.8B model. Small models do not always follow the requested output format exactly.

We expected structured fields such as:

triggered=true
reason=person remained in the region

However, the model sometimes returned variations such as:

trigger

or unstructured key-value lines.

Our original parser did not recognize those responses. It defaulted the verdict to false, meaning the full system appeared healthy while silently escalating zero events.

There was no crash and no obvious error. The pipeline simply became suspiciously quiet.

We fixed the problem by:

Building a defensive parser that accepts multiple valid output shapes
Adding explicit parse-failure telemetry
Logging rejected and malformed verdicts
Treating an unusually low escalation rate as a health warning
Testing against the actual local model rather than only mocked outputs

This taught us that model-output validation is not optional, especially when using small models in an automated pipeline.

Challenge 3: Securing the Camera-to-Edge Boundary

A surveillance system that is easy to spy on would be worse than no surveillance system at all.

During the final development stage, we hardened the camera-to-laptop connection.

Originally, the camera’s pairing QR code contained the same credential used by the edge bridge to communicate with the cloud. We redesigned this so that the camera now receives only a one-way secret derived using HMAC-SHA256.

As a result, nothing stored on the camera can authenticate directly against the cloud.

We also added:

A nonce-based HMAC challenge before the bridge trusts a camera
Loopback-only binding for the local preview server
Frame-rate and byte-rate limits
Message-size limits
Removal of credential-shaped values from firmware health reports
Safe servo boundaries
Permission checks for every camera action

The security changes did not alter what the demo looked like, which was exactly the goal. The protections should work without adding friction for the user.

Privacy by Design

Privacy was part of the architecture from the beginning rather than an additional feature added later.

Continuous video remains on the local network. The edge gateway processes the footage and filters routine activity. The cloud receives only selected evidence associated with candidate events.

This selective-upload approach provides several benefits:

Reduced bandwidth consumption
Lower cloud-inference cost
Less unnecessary exposure of private footage
Clearer evidence-retention controls
Better operation under weak network conditions

Access to cameras, events, recordings, and agent tools is restricted to the authenticated user. Device commands are permission-checked, short-lived credentials are used where possible, and every approved tool call is recorded.

Users remain in control of:

Which cameras are connected
Which monitoring rules are active
Which evidence is stored
How long evidence is retained
Which actions the assistant is allowed to request

Evaluation and Results

We wanted to support our edge-AI claims with measurable results.

Our benchmark harness drives the real pipeline, including object detection, agent filtering, local Qwen triage, clip recording, and event routing. The simulated component is the network transport, which is replaced by a byte-counting stub.

The benchmark used:

AMD Ryzen 7 5800U laptop
CPU-only inference
Simulated VGA camera input at 15 FPS
YOLO26-nano for stage-one detection
Qwen3.5 0.8B through Ollama for local triage
Qwen 3.7 Plus for cloud verification

Measured Results

Stage-one detection latency: p50 of approximately 114 ms per frame
Local triage cost: approximately 350 local-model tokens per rejected candidate
Cloud-model cost for locally rejected events: zero
Backend test suite: 148 tests passing
Flutter analysis: clean
Continuous video: remains local
Cloud upload: limited to candidate-event metadata and selected evidence

We measured the edge pipeline and cloud verifier separately. We intentionally do not claim a quantified false-positive reduction because we have not yet completed a labelled comparison against a threshold-only baseline.

We also built a zero-hardware demo mode so judges can experience the complete workflow, including real Qwen 3.7 Plus verification, without needing an ESP32 camera.

Resilience Under Weak and Offline Networks

The system was designed to degrade gracefully rather than stop working when connectivity is lost.

During a network interruption:

The ESP32-S3 continues capturing frames
The edge gateway continues running local detection
Previously compiled rules continue operating
Candidate events are marked as pending
Selected evidence is buffered locally
Pending events can be synchronized when connectivity returns

Cloud-based contextual verification may become temporarily unavailable, but deterministic local monitoring continues.

This separation allows the system to shift from full contextual reasoning to local rule-based monitoring without becoming unusable.

What We Learned

Put the Model Behind a Contract

Qwen can propose actions, but it should never directly operate hardware. Tool schemas, backend validation, movement limits, permissions, and audit logs must sit between the model and the physical device.

Validate What Models Actually Produce

A model’s output format is not guaranteed simply because a prompt requested it. Parsers must handle variation, detect malformed outputs, and expose suspiciously quiet failure modes.

Design for Degraded Conditions

Offline behavior cannot be an afterthought for an edge system. Local rules, buffered events, fallback compilers, and reconnection logic must be designed from the start.

Keep Privacy Boundaries Architectural

Privacy is stronger when raw footage does not need to leave the local network in the first place. Selective upload is more meaningful than relying only on policy or encryption after continuous footage has already been centralized.

Integration Is Harder Than Individual Features

Each component worked independently much earlier than the complete system worked end to end. The most difficult work involved synchronization, authentication, transport reliability, data formats, and failure handling between layers.

Potential Real-World Impact

Erlang AI Vision is designed for homes, small businesses, and privacy-sensitive environments where continuous cloud surveillance would be expensive or intrusive.

Potential use cases include:

Monitoring restricted entrances after business hours
Detecting unusual activity around a home
Notifying caregivers about safety-related situations
Monitoring stock rooms or loading areas
Identifying repeated activity rather than isolated motion
Providing searchable summaries of security events

Natural-language configuration makes monitoring more accessible to users who do not understand computer-vision settings.

At the same time, edge-first filtering reduces unnecessary cloud processing and keeps continuous footage within the user’s own environment.

What Comes Next

The laptop edge bridge proved that the architecture works, but it is still a prototype platform.

Our next steps are:

Move local inference to Jetson- or RK3588-class edge hardware
Improve multi-camera coordination
Add richer offline reasoning
Create a labelled evaluation dataset
Measure false-positive reduction against threshold-only detection
Improve local-model output reliability
Add configurable evidence-retention policies
Support more complex multi-step investigations
Strengthen device provisioning and hardware identity
Optimize GPU-assisted local triage

A dedicated local AI gateway could eventually manage several cameras in one home or business while maintaining the same edge-first privacy boundary.

Limitations and Responsible Use

Like any AI-assisted surveillance system, Erlang AI Vision can produce false positives, miss relevant activity, or interpret incomplete evidence incorrectly.

The platform should support human judgment rather than make automatic punitive decisions.

Responsible deployment requires:

Informed consent
Appropriate camera placement
Clear monitoring policies
Evidence-retention controls
Human review for serious decisions
Regular testing for false positives and missed events
Strict permission boundaries for camera actions

The goal is not to give a model unchecked control over a surveillance system.

Conclusion

Traditional CCTV is good at recording what happened. The next step is understanding which events deserve attention.

Erlang AI Vision turns a simple instruction such as:

“Alert me if someone lingers near the rear entrance after closing.”

into an auditable workflow that detects activity locally, gathers evidence safely, verifies context with Qwen, and alerts the user selectively.

The project taught us that building an agentic physical system requires more than connecting a model to a camera. It requires clear contracts between AI and deterministic software, secure communication across private networks, resilient offline behavior, careful output validation, and privacy boundaries built into the architecture.

Our aim is to make surveillance more useful without making it more intrusive:

Local perception. Selective cloud reasoning. Explicit safety constraints. Graceful offline operation.

Built With

alibaba-cloud
arduino
c++
caddy
dart
dashscope
docker
esp32
fastapi
firebase
flutter
freertos
mcp
ollama
opencv
platformio
postgresql
python
qwen
sqlalchemy
sqlite
websockets
wokwi
yamnet
yolo

Erlang AI Vision: Agentic Edge Surveillance with Qwen