Bijou

BIJOU is an assistant CISO real-time AI live agent powered by Gemini 2.0 Flash Live that translates your voice, analyzes your screen, and whispers tactics only you see. Be yourself in any language.

Comment

Inspiration

I'm a French speaker. My English is decent I understand most things but there's a difference between understanding and being yourself.

Every pitch meeting, every client call, every conference abroad: I was performing. Not selling. Not connecting. Performing. Translating in my head, second-guessing every word, missing the rhythm of the conversation. While my English-speaking competitors were freestyling, I was processing.

I tried every translation tool on the market. They all work the same way: you talk, you wait, it translates, the other person waits, they respond, you wait again. By the time the tool finishes, the conversation is dead. Good luck closing a deal with a 3-second lag after every sentence.

What I wanted didn't exist: a live agent that sits inside the conversation not after it. Something that translates my voice as I speak, so smoothly that the person on the other end doesn't even realize I'm not speaking their language natively. Something that lets me be me my energy, my humor, my sales instincts in any language.

So I built BIJOU.

What it does

BIJOU is a real-time AI live agent powered by Gemini 2.0 Flash Live that translates your voice, analyzes your screen, and coaches you all during the conversation, not after.

🗣️ Real-time voice translation I speak French, my client in Tokyo hears Japanese. My client in Madrid hears Spanish. 21 languages including Wolof, Arabic, Mandarin, Portuguese. Sub-200ms latency. No awkward pauses. No "please wait while I translate." Just conversation.

👁️ Live screen analysis BIJOU sees what's on screen. Share your AWS console during a client call? She detects misconfigurations, flags compliance gaps, generates production-ready Terraform remediation code. In a terminal overlay. Deployable, not pseudocode.

🕵️ Stealth channel While you're speaking publicly, BIJOU simultaneously whispers tactical intelligence _only to your screen. "Client claims SOC2 compliance. Screen shows unencrypted RDS instance flag this." Your client never sees it. Zero tells.

📊 Smart pitch deck Slides change automatically based on conversation context. Client asks about architecture → the right slide appears. No manual clicking.

✉️ Instant post-meeting Click Stop → summary, action items with assignees and deadlines, follow-up email ready to send. Before the client closes their laptop.

The real endgame? I'm turning BIJOU into a universal translator for:

  • Video conferences Zoom, Meet, Teams BIJOU joins as your live translation layer
  • Earbuds for pitches real-time whispered translation directly in your ear during in-person meetings
  • Travel airports, hotels, restaurants so language stops being a barrier and everyone can truly be themselves and be understood

How we built it

Architecture: React → AudioWorklet → WebSocket → Python asyncio → Gemini 2.0 Live API → 10 Function Calling tools → dual-channel response

  1. Audio Capture: Custom pcm-processor.js AudioWorklet encodes directly to 16-bit PCM at 16kHz not the standard getUserMedia with its 200-400ms of buffering lag
  2. Video Capture: Screen/camera frames at 1 FPS as JPEG, sent alongside audio via WebSocket
  3. Backend Bridge: Python asyncio WebSocket server with two permanently-running concurrent tasks receiver_task (Gemini → frontend) and sender_loop (frontend → Gemini) true full-duplex, no polling
  4. Gemini 2.0 Flash Live: client.aio.live.connect() with bidiGenerateContent real bidirectional audio+vision+text+function calling in a single persistent session
  5. 10 Function Calling Tools: Each triggers a specific frontend action (highlight vulnerability, generate code, change slide, write email, stealth whisper)
  6. Dual-channel routing: trigger_stealth_whisper routes to host's WebSocket ONLY never broadcast
  7. NGINX: Reverse proxy + WebSocket upgrade + static file serving in a single container
  8. Google Cloud Run: Single Docker container, GCP Secret Manager for API key injection, zero-ops deployment
  9. Terraform IaC: Full infrastructure-as-code for reproducible Cloud Run deployment

Wire protocol (typed JSON over WebSocket):

  • bijou_response → transcript + translated audio
  • stealth_alert → private whisper to host only
  • ui_highlight → red bounding box on detected vulnerability
  • ui_terminal → terminal overlay with typewriter code animation
  • slide_change → auto-navigate presentation
  • followup_email → post-meeting email
  • summary_update / action_item → live intelligence panel

Challenges we ran into

  1. asyncio full-duplex without deadlock The Gemini Live API requires truly simultaneous send/receive. Python asyncio needed two permanently-running concurrent tasks with proper cancellation and backpressure. A single coroutine would deadlock.

  2. nginx HTTP/1.0 → WebSocket 502 nginx defaults to HTTP/1.0 for internal proxy pass. WebSockets require HTTP/1.1. Had to add proxy_http_version 1.1 to every relevant location block including health checks, not just /ws.

  3. Model naming in the Live API gemini-2.0-flash works for REST but raises "not supported for bidiGenerateContent" in the Live API. The correct identifier is models/gemini-2.0-flash-live-001 a Live-specific endpoint.

  4. Stealth channel isolation The dual-channel guarantee means stealth whispers MUST route only to the host's WebSocket. Implementing per-role routing inside the room manager while keeping generic fan-out for everything else required careful session state design.

  5. AudioWorklet latency Standard getUserMedia introduces 200-400ms of startup lag. Building a custom AudioWorklet that encodes PCM directly and sends synchronous chunks via WebSocket brought latency under 200ms.

  6. Staying focused on the human problem The hardest challenge was resisting the urge to build a cybersecurity tool and remembering why I started: I just wanted to stop feeling like a foreigner in my own meetings.

Accomplishments that we're proud of

  • Sub-200ms voice translation latency in production fast enough that conversations feel natural
  • The stealth channel works flawlessly Gemini detects a claim, flags it privately, I pivot the conversation before the client finishes their sentence
  • All 10 tools work coherently in a live session without conflicts
  • Post-meeting email is ready before the call disconnects
  • Built and deployed a production multimodal live agent in a single Docker container, $0 Kubernetes overhead
  • My mom tested it. She spoke Wolof. The client heard English. She cried.

What we learned

  • Gemini 2.0 Live API is fundamentally different from all prior AI APIs you're maintaining a persistent bidirectional connection, not making requests. The mental model shift matters.
  • 1 FPS vision frames are sufficient Gemini doesn't need continuous video to detect an open S3 bucket
  • Session affinity on Cloud Run is non-negotiable for WebSocket services
  • bidiGenerateContent has its own model registry generic aliases don't work
  • The best technology disappears. When the translation is fast enough, you forget it's there. That's when you start being yourself again.

What's next for BIJOU

  • Earbuds integration Bluetooth earpiece for in-person pitches with real-time whispered translation
  • Conference mode Multi-participant rooms where everyone speaks their native language and hears theirs
  • Video conference plugins Zoom, Google Meet, Teams integration as a translation layer
  • Travel mode Lightweight mobile version for airports, hotels, restaurants
  • More languages Expanding beyond 21 to cover every language where someone feels limited by translation
  • Voice cloning Your translated voice sounds like you, not a robot

Built With

Share this project:

Updates