Scout: The Always-On, Private Personal AI Assistant

"Scout: the always-on personal AI you can trust to act for you, never as you. Dual NVIDIA OpenShell sandboxes keep your data away from the web, so a hijacked page can’t make it share, send or spend."

Inspiration

We all want a genuine personal AI assistant: an always-on companion that handles emails, calendars, files, and everyday errands. The spark for Scout came from an office discussion with colleagues about autonomous web bots like Grok. While everyone loved the idea of proactive AI, the room was unanimous on one point: nobody actually trusted them. High-profile security failures, such as autonomous bots tricked via indirect prompt injection into leaking sensitive context and publishing user credit card details to the public, proved that this fear was well justified. Existing agentic systems collapse execution and security into a single runtime environment. When an AI has simultaneous access to your private context and the untrusted web, any malicious webpage, compromised email, or hidden prompt injection can hijack the model. A single malicious site can easily instruct the agent to ignore its original rules, harvest private data, or execute unauthorized actions on your behalf. This fundamental security flaw inspired Scout's dual-agent architecture. To neutralize the threat, Scout strips all private context away from external web interactions by offloading browsing to a dedicated second agent. Operating inside an isolated sandbox, this web scout runs with zero private memory, zero credentials, and zero access to your local environment. Even if a malicious website takes full control of the web agent, the blast radius is reduced to zero because there is simply no private data available to share, send, or spend.

What It Does

Scout is an always-on, cross-platform personal AI (Desktop, Web, and Mobile) that acts on your behalf while keeping you firmly in control:

Dual-Sandbox Zero-Trust Architecture: Scout runs as two separate agents in isolated NVIDIA OpenShell environments:
- User Agent (scout-user): Holds read-only access to your private files, email, calendar, and persistent memory. It has zero access to the open web.
- Browser Agent (scout-browser): Navigates the public web through a strict egress proxy. It holds zero personal data.
- The two communicate strictly through a typed contract validated by NVIDIA NeMo Guardrails and NemoClaw. Even a fully compromised webpage cannot cross the sandbox barrier.
"Acts for you, never as you": Any consequential write—spending money, sending emails, or sharing sensitive personal details—generates a deterministic human-in-the-loop approval card that the model itself cannot dismiss.
Location Awareness & Commute Convergence: Using on-device Apple Maps (MapKit MKDirections), Scout continuously evaluates real-time traffic ETAs to automate everyday tasks:
- Commute-Timed Deliveries: Scout synchronises grocery delivery orders so packages arrive exactly as you pull into your driveway.
- Arrival-Relative Reminders: Triggers dynamic notifications "5 minutes before reaching office" based on live traffic, without your raw GPS coordinates ever leaving your device.
Persistent Memory & Reusable Skills: Cross-session memory remembers your habits, preferences, and people, while an autonomous skill engine writes and refines reusable procedures on the fly.
Open-Source NVIDIA Models on Nebius: Primary inference runs on NVIDIA Nemotron (nvidia/Nemotron-3_5-Lightning) via Nebius Token Factory, providing high-throughput, serverless token inference.

How We Built It

Hermes Agent Framework: Built on Nous Research’s Hermes Agent architecture, utilizing its modular multi-turn execution loop, extensible skills library (skills/), cron background scheduler (cron/), and SQLite session database.
NVIDIA OpenShell & Sandboxing: Each agent is partitioned into its own OpenShell policy container. Egress network filters drop private network IPs and cloud-metadata addresses.
NVIDIA NeMo Guardrails & NemoClaw: Incoming web content is sanitized as untrusted data. NeMo Guardrails enforces strict type checks on bridge outputs.
Nebius Serverless & Token Factory: Scout routes reasoning and tool calling to open-source NVIDIA Nemotron 3.5 Lightning hosted on Nebius Token Factory with pay-per-token serverless inference. The backend is packaged into an OCI container with target deployment configurations for Nebius Serverless AI Endpoints.
Spatial Tracking Engine: Built native bindings to Apple Maps MapKit MKDirections for real-time traffic computations. Driving estimates are calculated locally and memoized, preserving complete spatial privacy from cloud LLMs.
Multi-Surface Client: An Electron desktop application with a real-time JSON-RPC daemon, paired with a web dashboard and responsive mobile companion.

Challenges We Faced

Agent Harness & Safety Guardrails: Building a robust harness to keep autonomous agents bounded without sacrificing utility proved difficult. Setting up tight safety guardrails required constant tuning to prevent models from bypassing operational limits.
Local Inference & Hallucinations: Running on local hardware required using smaller open-weight models. These models are far more prone to hallucinations and instruction drift, making reliable structured output generation a major hurdle.
Memory Architecture: Designing an isolated memory layer that provides persistent context to the core agent without polluting state or creating memory leaks across executions.

What We Learned

Harness Engineering is Paramount: The quality of the agent harness and safety guardrail pipeline matters just as much as the underlying LLM. A strong harness can make smaller local models perform reliably.
Adversarial Inter-Agent Protocols: If two agents communicate in plain English, a prompt injection attack on the external agent can easily propagate to the core agent. We learned to build structured, sanitized inter-agent protocols that pass complete payload context while filtering out malicious instructions.
Real-Time Orchestration & Monitoring: Implementing active agentic orchestration is essential. Having the local Core Agent monitor the internet-facing Web Scout in real time allows it to track task progression, catch execution drift early, and abort operations if the sub-agent falls victim to a hijacked page.

What's Next for Scout

Multimodal Vision & Screen-Aware Cursor Agent: Introducing vision capabilities to allow Scout to see your screen, explain visual content on demand, and assist with desktop-level workflows.
Generative Visual Explanations: Enabling Scout to dynamically generate or synthesize images and videos to explain complex tasks and concepts visually.
OS-Level PC Interactions: Expanding Scout's capability matrix to securely perform desktop and operating system tasks on your behalf.
Advanced Memory & Local Model Tuning: Refining the long-term memory layer and fine-tuning smaller, highly capable open-source models specifically optimized for low-latency local inference.