Astro-Sentinel

Privacy-Preserving Physical AI Droid

Comment

Inspiration

As AI moves beyond screens and into our homes, desks, and workplaces, a critical question emerges: Can an AI understand the physical world without sending our private surroundings to the cloud?

Most camera-enabled AI applications rely on remote processing, potentially exposing images of personal spaces, people, and sensitive information. I wanted to explore a different approach—one where intelligence can interact with the physical world while keeping visual perception local.

Inspired by the playful personality of droids like R2-D2, I created Astro-Sentinel, a privacy-first physical AI companion that sees locally, reasons selectively, and responds physically.

What it does

Astro-Sentinel is an embodied AI companion that senses nearby objects, interprets them on-device, uses cloud AI for reasoning, and expresses its decisions through physical interactions.

Here's how an interaction works:

  • Senses the world: A distance sensor detects when an object enters the interaction zone, triggering the USB camera.
  • Sees locally: An on-device Vision-Language Model (Qwen 3-VL-8B) understands the environment & identifies objects without sending the image to a remote vision service.
  • Reasons through Nebius: Only the sanitized object label and distance measurement are sent to NVIDIA Nemotron 3.5 Lightning through Nebius Token Factory.
  • Responds physically: NVIDIA Nemotron 3.5 Lightning returns feedback which drives expressive LED animations, droid-like sound effects, and haptic feedback.

The temporary image is deleted after local inference. A dedicated camera indicator makes camera activity visible, while the device maintains responsive behavior when cloud connectivity fails.

Astro-Sentinel turns AI from a screen-based interaction into a tangible, multimodal experience—without making raw camera images part of the cloud reasoning request.

How I built it

I combined embedded hardware, on-device multimodal AI, and cloud-based reasoning in a two-tier edge-to-cloud architecture.

Hardware and physical interaction

I use the Arduino VENTUNO Q as the core platform, paired with an Arduino USB camera, a Modulino distance sensor, a Modulino buzzer, and a Modulino vibration module. The onboard 8×13 LED matrix displays the device's state and expressive droid faces.

On-device perception and privacy

Arduino App Lab's Vision-Language Model Brick runs Qwen 3-VL-8B locally on the board's Qualcomm NPU. Captured frames are held in temporary local storage and removed immediately after inference.

Cloud reasoning with NVIDIA Nemotron via Nebius

I use Nebius Token Factory to access NVIDIA Nemotron 3.5 Lightning through an OpenAI-compatible API. The model receives the sanitized object label and distance metric, then returns a constrained interaction state: HAPPY, CURIOUS, or CONFUSED.

Real-time orchestration

I developed a Python orchestrator to handle sensing, local inference, cloud requests, and session state. Embedded C++ firmware controls the LED matrix and physical actuators.

I implemented spatial hysteresis, asynchronous processing, session invalidation, and an offline fallback to help maintain responsive behavior despite sensor-boundary fluctuations, network delays, or disappearing objects.

The result is a system that connects physical perception, AI reasoning, and expressive hardware while minimizing the information shared with the cloud.

Challenges I ran into

Balancing rich perception with privacy

A vision model needs access to image data to recognize objects, but I wanted to avoid transmitting that data to the cloud. I addressed this by separating local visual perception from cloud reasoning and introducing a sanitization layer before network transmission.

Keeping the physical interaction responsive

Cloud inference can take longer than sensor polling. Running everything synchronously would risk freezing the interaction loop. I separated background AI work from continuous sensor polling so the hardware can remain responsive while reasoning is in progress.

Preventing unstable state transitions

Objects may hover around the sensing threshold, causing repeated activation and deactivation. I introduced spatial hysteresis: an object triggers interaction at 300 mm or closer, while the session resets only after the object moves beyond 360 mm.

Handling asynchronous responses

An object might disappear while a cloud request is still running. I added session-generation tracking so that late responses can be discarded rather than triggering reactions to an object that is no longer present.

Failing gracefully

A physical AI device should not become unresponsive just because Wi-Fi or an API is unavailable. I designed an offline fallback that allows Astro-Sentinel to return to a local CONFUSED state while maintaining hardware responsiveness.

These challenges reinforced an important lesson: successful physical AI depends not just on model intelligence, but also on reliability, timing, and clear interaction design.

Accomplishments that I'm proud of

  • A complete physical AI interaction loop: I connected real-world sensing, on-device vision, cloud reasoning, and physical responses in one system.
  • Privacy by architectural design: Raw images are processed locally and deleted after inference, while cloud reasoning receives only sanitized text and distance metadata.
  • Visible, expressive feedback: I integrated the LED matrix, buzzer, and vibration module to turn abstract AI states into recognizable physical interactions.
  • Robustness beyond the happy path: I addressed real-world edge cases through spatial hysteresis, asynchronous task isolation, session invalidation, and an offline fallback.
  • An edge-to-cloud AI architecture: I combined local Qwen 3-VL-8B inference with NVIDIA Nemotron through Nebius Token Factory to demonstrate how on-device perception and cloud reasoning can complement each other.

Most importantly, I'm proud to explore a different model for ambient AI: one that can be physically interactive without requiring raw visual information to be sent to a cloud vision service.

What I learned

physical AI is as much about system design as it is about model capability.

A capable vision model is only one part of the experience. Sensor thresholds, asynchronous execution, hardware feedback, stale responses, and network failures all shape whether an embodied system feels intelligent and dependable.

There is value in separating perception from reasoning. By running visual recognition locally and sending only a restricted description to the cloud, I can explore different models for cloud-based reasoning without making raw image transmission a requirement of the architecture.

What's next for Astro-Sentinel

Astro-Sentinel is a starting point for privacy-conscious physical AI. My next steps focus on making the system more capable by experimenting with different models.

Astro-Sentinel: It sees locally. It reasons selectively. It responds physically.

Built With

Share this project:

Updates

Submission history