-
-
This is My main page.
-
This is my Architecture Diagram.
-
This is my ci/cd part using Github-action.
-
This is My GCP VM.
Inspiration
I have always been fascinated by the rapid advancements in AI and wanted to push the boundaries of what the Gemini API could do. When the #GeminiLiveAgentChallenge was announced, it provided the perfect catalyst. I wanted to build something beyond standard text chatbots—a tool that actively assists developers by "seeing" and "hearing" in real-time.
What it does
Nexus acts as an elite, on-demand cybersecurity expert (Live Agent). It monitors a developer's screen during coding or system configuration and listens to their voice commands. By analyzing the visual context, it instantly identifies potential security vulnerabilities, misconfigurations, or hardcoded secrets, and provides immediate, natural voice feedback—acting as a real-time pair-programming security auditor.
How we built it
I engineered a full-stack, multimodal architecture: Frontend: Built with Next.js, capturing screen video frames and microphone audio via WebRTC, and handling downsampling for audio streams. Backend: Developed using NestJS to act as a robust WebSocket server. It serves as a secure proxy to manage the bidirectional data stream. AI Integration: Integrated Google's Gemini Live API (BidiGenerateContent) to process the interleaved visual and audio data. Infrastructure: The entire system is containerized with Docker and deployed on a Google Cloud Compute Engine VM, with a fully automated CI/CD pipeline via GitHub Actions.
Challenges we ran into
Integrating a true bidirectional multimodal stream was incredibly challenging. Initially, syncing the continuous audio/video chunks with the Gemini API caused connection drops. Another major hurdle was processing the audio playback on the frontend—handling browser autoplay policies, converting 32-bit float audio to 16-bit PCM, and managing Endianness to ensure the AI's voice played smoothly without static noise. Additionally, navigating Google Cloud deployments and configuring Docker alongside GitHub Actions for the first time presented a steep learning curve.
Accomplishments that we're proud of
I am incredibly proud of successfully bridging the gap between raw browser media streams and the Gemini Live API to create a seamless, real-time conversation. Seeing the AI accurately read the code from a screen share and immediately speak back with a security warning was a massive "Wow" moment. Overcoming the DevOps challenges to achieve a fully automated GCP deployment was also a major personal victory.
What we learned
This project was a massive learning experience. I gained deep, hands-on knowledge of WebSocket communication, WebRTC media buffer processing, and the intricacies of the Gemini multimodal API. Furthermore, I learned essential DevOps practices, successfully navigating Google Cloud Platform (VM instances, Firewall rules) and setting up CI/CD pipelines with Docker and GitHub Actions for the first time.
What's next for Real-time security auditor assistant
The next step is to enhance the agent's accuracy by fine-tuning its system instructions to detect more complex, zero-day vulnerabilities. I also plan to add features like highlighting specific vulnerable lines of code directly on the user's screen and integrating it with IDE plugins for an even more seamless developer experience.
Log in or sign up for Devpost to join the conversation.