PPISec: Semantic Firewall for VLA Robotics

PPISEC. A brain held by a robot arm and a shield in the middle. Imagine a LLM interacting with a robot and security protecting the robot.
PPISEC Logo with a connection between a brain(LLM) and robot with a shield between, acting as a prompt injection protection.

Inspiration

The rapid evolution of Vision-Language-Action (VLA) agents has created a shift from digital interactions to physical consequences; however, this transition introduces a terrifying new attack vector. While researchers have extensively studied text-based prompt injection, the physical world introduces a far more deceptive threat through Physical Prompt Injection Attacks (PPIA). An adversary can manipulate an embodied agent simply by placing a "Trojan sign" or a malicious sticky note in the robot's field of view. We were inspired to build PPISec because we realized that current robotic brains are dangerously gullible; they lack a dedicated prefrontal cortex to filter malicious environmental cues. We wanted to create a system that treats every perception as a hypothesis that must be validated against a rigorous safety manifest before it can be translated into mechanical motion.

What it does

PPISec acts as a high-performance, AASL-compliant middleware that sits between a robot’s visual brain and its mechanical muscles. It implements a four-stage security pipeline designed to detect and neutralize physical attacks in real time; the system validates every proposed movement against a strict policy manifest. Through our Agent Glass dashboard, users can see the robot’s "latent thoughts" in a 3D environment, visualizing exactly why a specific action was blocked by the firewall. By cross-referencing audio instructions with visual intent, we ensure that the robot only executes commands that are consistent across all sensory modalities.

How we built it

The architecture is entirely distributed and platform-agnostic, allowing it to scale from edge devices to cloud clusters. The core of the system is the Firewall Governor, which was built using FastAPI to handle high-concurrency security requests; it orchestrates a four-stage pipeline that evaluates every intent in real time. We utilized vLLM to serve the Qwen2-VL-7B model, providing the visual reasoning necessary to identify environmental hazards; we then developed Agent Glass using Next.js and Three.js to provide a 3D observability sandbox. To ensure deterministic safety, we integrated a logic engine that monitors spatial invariants; this engine evaluates whether the agent’s proposed trajectory satisfies a specific safety formula $\phi$ using Linear Temporal Logic (LTL):

$$\phi = \square (\text{agent_state} \in \mathcal{S}_{safe})$$

This ensures that for all time steps, the agent must remain within the globally defined safe state space $\mathcal{S}_{safe}$.

Challenges we ran into

Our most significant hurdle was the Multimodal Conflict Resolution (MCR) problem; specifically, the system must decide how to react when a visual cue contradicts a verbal instruction. We had to develop a semantic weighting algorithm that calculates a combined confidence score $C_{final}$ to prevent the agent from being paralyzed by conflicting inputs; the confidence is calculated as:

$$C_{final} = \frac{\sum_{i \in {v, a}} w_i \cdot \text{sim}(\text{intent}_i, \text{policy})}{\sum w_i}$$

In this formula, $w_i$ represents the trust weight of the modality; $\text{sim}$ represents the cosine similarity between the perceived intent and the allowed policy manifest. Additionally, we struggled to keep the end-to-end latency below fifty milliseconds. We solved this by implementing a Radix Tree for $O(1)$ policy lookups and optimizing our sentence-transformer models to run on edge-compatible runtimes.

Accomplishments that we're proud of

We are incredibly proud of achieving a system that can detect "hidden" malicious intent that traditional computer vision would completely miss. Successfully integrating Linear Temporal Logic into a real-time AI pipeline feels like a massive win for robotic safety; it provides a mathematical guarantee that the AI cannot violate core safety rules. Seeing the Agent Glass dashboard come to life with real-time 3D visualizations was the moment we knew we had built a truly innovative observability tool.

What we learned

This project taught us that robotic safety is not a single feature; it is a multi-layered governance problem. We gained deep insights into the Autonomous Agent Safety Layer (AASL) framework; we also learned how to bridge the gap between high-level semantic reasoning and low-level physical execution. We discovered that while Large Vision-Language Models are powerful, they require a deterministic "watchdog" to ensure they do not hallucinate unsafe actions in critical environments.

What's next for PPISec

The next step is moving PPISec from our mock environment onto physical hardware like the Unitree Go2 or a Boston Dynamics Spot. We plan to expand our policy engine to support more complex collaborative tasks; we also want to explore decentralized security nodes that can share attack signatures across a fleet of robots. PPISec is the foundation for a world where we can finally trust the autonomous machines moving around us.