-
-
hiBOB
-
Architecture
Inspiration
Phishing links, malicious QR codes, data breaches — most people encounter these threats daily without realizing it. We wanted to build an AI agent that acts as a personal security guardian: always listening, always watching, ready to analyze any threat in real time just by talking to it.
What it does
hiBOB is a multimodal live AI security agent. You talk to it naturally, and it:
- Analyzes URLs, IPs, domains and files with VirusTotal API v3
- Scans QR codes from your gallery or live camera before you click anything
- Checks if your passwords have been leaked using Have I Been Pwned with k-Anonymity (your password never leaves the server in plain text)
- Generates cryptographically secure passwords with entropy calculation
- Guides you step-by-step through any task on your phone as a copilot, using your screen or camera as context
- Analyzes your surroundings in real time via rear camera — detecting URLs, QR codes and threats in the physical world
How we built it
- Flutter (Android/iOS) — camera, microphone, PCM audio streaming, security overlays
- NestJS (TypeScript) backend hosted on Google Cloud Run
- Gemini Live API (
gemini-live-2.5-flash-native-audio) via Vertex AI — real-time audio + vision + tool calling - Socket.IO — bidirectional WebSocket for audio, frames and events
- Firebase Auth + Firestore — user authentication and preferences
- CI/CD: GitHub Actions → Artifact Registry → Cloud Run (automated on every push)
Challenges we ran into
- Gemini Live session stability: keeping the session alive through reconnections while maintaining full context was the hardest problem. We built transparent reconnection logic that preserves the conversation state across socket drops.
- Manual activity detection: Gemini Live's automatic activity detection would interrupt image analysis mid-processing. We had to carefully control when to pause audio input during file uploads.
- QR scanning reliability: A single jsQR pass wasn't enough. We built a multi-strategy pipeline (original → contrast boost → grayscale threshold) with 4 rotation attempts each to handle real-world QR codes.
What we learned
That Gemini Live is genuinely capable of acting as a real security agent — not just answering questions, but taking actions, calling tools, and reacting to visual input in real time. The multimodal + live combination unlocks a class of applications that simply weren't possible before.
What's next
- Windows/macOS desktop version for enterprise security monitoring
- Browser extension integration for real-time phishing detection
- Organization-wide threat intelligence dashboard
Log in or sign up for Devpost to join the conversation.