NIBO

• 

  Project Architecture

Inspiration

NIBO was born from the hundreds of hours I spent as a customer service representative, listening to the voices of people who had just lost their life savings to a scam. In those calls, I realized a fundamental flaw: current security is entirely reactive. It blocks a malicious URL only after the user has been convinced to click it.

I saw that social engineering doesn't happen in the code; it happens in the air—through conversation, urgency, and human pressure. I built NIBO to be the proactive sentinel I wish those people had. By moving the security perimeter from the network layer to the ambient environment, we’ve created an AI that doesn't just automate your life, but intercepts malicious intent before it ever becomes a transaction.

What it does

NIBO is a Safe-Traversal Autopilot for the web. It uses voice-actuated AI to execute complex browser workflows through a One-Shot Heuristic Engine that records AI actions as local macros. Simultaneously, its Ambient Sentinel monitors environmental audio (like phone calls) to detect and intercept social engineering scams in real-time before they reach the browser's DOM.

How we built it

We architected a distributed system using:

Gemini 2.5 Flash Multimodal Live API for sub-second native audio and visual inference. Python/FastAPI on Google Cloud Run to handle high-frequency WebSocket streams. Chrome Extension (React/Vite) with an offscreen.html pipeline for continuous PCM audio capture and high-fidelity DOM manipulation.

Challenges we ran into

Sub-Second Latency: Optimizing the binary WebSocket pipeline to ensure a scam warning can "barge-in" and interrupt the conversation immediately. Zero-Trust Privacy: Implementing a local PII Redaction Pass to scrub credit card numbers and sensitive data at the edge before cloud egress. DOM Noise: Filtering dynamic website clutter into semantic ARIA landmarks for reliable model reasoning.

Accomplishments that we're proud of

Zero-Inference Replay: Successfully serializing non-deterministic AI behavior into stable, locally-persistent macros that run with zero delay. Ambient Security Intercept: Engineering a system that can detect verbal pressure tactics and proactively lock the user's browser UI. Production Stability: Building a resilient, containerized backend capable of managing multi-modal streams at scale.

What we learned

Security is Holistic: Real-world risk isn't just in the code; it’s in the ambient conversation and user context. Hybrid Intelligence: The best AI experience combines the flexibility of LLMs in the cloud with the speed and privacy of local execution. Silence is Power: A security agent must be silent by default but authoritative when a threshold is breached.

What's next for NIBO

Adaptive Safety Gates: Dynamically adjusting risk thresholds based on global threat telemetry. OS-Level Protection: Expanding NIBO's ambient sentinel beyond the browser to provide system-wide social engineering protection. Self-Healing Workflows: Using feedback loops to automatically update macros as website structures evolve.

Built With

• cloudrun
• docker
• fastapi
• gcp
• react
• vertex