AgentNode: AI Vision for Everyday Devices

• 

• 

• 

• 

Inspiration

AgentNode started with a simple question: can everyday devices such as smartphones and tablets become useful AI participants instead of remaining isolated cameras or sensors?

Many households need occasional awareness of what is happening at home, but continuous video surveillance can be expensive, invasive and dependent on proprietary hardware. We wanted to explore a more flexible approach: use devices people already own, give them programmable capabilities, and let an AI-powered backend interpret specific situations only when needed.

What it does

AgentNode turns a browser-enabled smartphone, tablet or computer into a Vision Node.

A user can:

• Assign a device to a home and room.
• Define a natural-language surveillance prompt.
• Configure a capture interval.
• Grant access to the device's rear camera.
• Review image captures and structured AI analysis.
• See whether the configured condition was detected.
• Optionally receive an email alert.

Each image is captured independently while the PWA is open and sent to the authenticated NestJS backend. GPT-5.6 analyzes the image using Structured Outputs and returns a description, confidence level, detected condition and optional alert message.

The current MVP focuses on vision-based vigilance, while the architecture is designed to support additional AgentNode capabilities in the future.

How we built it

AgentNode is an npm-workspaces monorepo built with Angular, NestJS, TypeScript, Firebase Authentication, Firestore and the OpenAI Responses API.

Angular provides the authenticated user interface and camera workflow. NestJS owns authentication, authorization, validation, persistence and communication with OpenAI. Firestore stores configuration and capture metadata, while the JPEG files remain in protected backend storage rather than in the database.

GPT-5.6 is used through a centralized backend prompt and Structured Outputs. The model produces analysis and logical decisions, but it never accesses hardware directly, writes to Firestore or controls devices. The backend validates the model output and remains the operational authority.

Codex was used throughout the build to inspect the repository architecture, implement bounded tasks, create shared TypeScript contracts, develop backend and frontend flows, add tests, diagnose issues and validate the final behavior.

Challenges we ran into

The main challenge was creating a useful camera experience without turning the MVP into a continuous video surveillance system. We had to carefully bound image size, capture frequency, storage paths, retries and OpenAI requests.

We also needed to preserve a strict separation between physical Devices, logical AgentNodes, backend orchestration and AI reasoning. This required explicit ownership checks, authenticated image access, centralized prompts and transactional Firestore operations.

Another challenge was making the feature work inside a browser while handling camera permissions, asynchronous capture, upload failures, analysis states and a clear user experience.

Accomplishments that we're proud of

We built a complete authenticated Vision Node flow using ordinary browser-enabled devices.

We are especially proud that the MVP:

• Uses GPT-5.6 for real multimodal analysis rather than as a decorative chatbot.
• Produces structured, validated results for every capture.
• Keeps image bytes out of Firestore.
• Protects captures through backend authentication and ownership checks.
• Supports configurable alerts by email.
• Avoids video streaming, Firestore polling and unnecessary infrastructure.
• Includes shared frontend/backend contracts, automated tests and implementation reports.
• Preserves a clean architecture for future AgentNode capabilities and orchestration.

What we learned

We learned that reliable AI products require clear boundaries around model authority. GPT-5.6 is excellent at interpreting a scene and producing structured reasoning, but the backend must validate its output and control what happens next.

We also learned that privacy and cost constraints should shape the architecture from the beginning. Independent captures, bounded intervals and protected storage make the MVP easier to understand, safer to operate and more affordable than continuous streaming.

Finally, using Codex as an engineering partner showed us how quickly a small team can move when repository exploration, implementation, testing, documentation and debugging are handled as one continuous workflow.

What's next for AgentNode

The next stage is to connect multiple AgentNodes through the existing orchestration architecture. Future versions can coordinate capabilities such as image capture, speech, status reporting and environmental sensing across several devices.

The long-term goal is a household network of reprogrammable AI nodes that can assist with safety, caregiving, object location, communication and routines—using existing devices while keeping the human owner in control.

Built With

• angular.js
• browser-camera
• codex
• firebase-authentication
• firestore
• gpt-5.6
• nestjs
• npm-workspaces
• openai-responses-api
• structured-outputs
• typescript
• web-apis