Inspiration
Unmanned Systems (UxS) and drone operations often rely on complex, clunky, and hands-on interfaces that require extensive training and split the operator's attention. In high-pressure tactical environments, speed and ease of communication are critical. We were inspired by the rapid advancements in Large Language Models to create a military-grade, hands-free interface. By combining voice recognition with AI, we wanted to build a system where operators can simply speak naturally to control a drone, much like communicating with a human pilot, drastically reducing cognitive load.
What it does
VDUX is a browser-based tactical drone command interface that allows operators to control a simulated military drone using natural voice commands or text input. It features:
- Voice Command Input: Utilizes the native Web Speech API to listen and transcribe operator commands in real-time.
- AI-Powered Parsing: Integrates Google Gemini 2.0 Flash to understand complex, free-form natural language (e.g., "Fly north and scan the perimeter at 40 meters").
- Live Tactical Map: A rich HTML5 Canvas simulation that renders the drone's position, heading, trail, restricted zones, and operational bases in real time.
- Real-time Telemetry: Dynamic readouts of altitude, speed, signal strength, battery life, pitch, and roll.
- Restricted Zone Enforcement: Safety protocols automatically reject commands that would send the drone into no-fly zones (like the Inner Compound or Armory).
How we built it
We built VDUX entirely as a zero-dependency, single-file application using plain HTML, CSS, and JavaScript so it can run seamlessly in any modern browser. The tactical UI was designed from scratch with a dark mode aesthetic, CRT scanlines, and custom metrics animations to simulate a high-tech military terminal.
For the core logic, we used HTML5 Canvas to build the physics and rendering engine for the drone's movement and map overlays. We integrated the browser's native Web Speech API to handle microphone input and speech-to-text. The transcribed text is then sent to the Google Gemini API, which acts as the intelligent brain of the system, extracting intent, target locations, and flight parameters from conversational language and mapping them to actionable system commands.
Challenges we ran into
One of the main challenges was parsing unstructured natural language into precise coordinate and altitude changes. People speak in unpredictable ways, so instead of using rigid regex patterns, ensuring the Gemini API prompt was engineered correctly to reliably return executed commands was a major hurdle.
Another challenge was building the physics and animation loop for the drone in HTML5 Canvas. We had to ensure the drone accelerated, banked (pitch/roll), and decelerated smoothly towards its target—rather than just snapping to the location—to make the simulation feel realistic.
Accomplishments that we're proud of
We are incredibly proud of the seamless integration of the Web Speech API with Gemini 2.0 Flash. Seeing the drone respond accurately to complex, multi-part voice commands in near real-time feels like magic and proves the concept of AI-driven piloting. We're also proud of the UI design—it genuinely looks and feels like a professional tactical control terminal.
What we learned
We learned a tremendous amount about prompt engineering for LLMs in a strict control-system context, moving away from chatbots and into functional system actions. We also deepened our knowledge of HTML5 Canvas for real-time rendering and state-based animation loops. Finally, working with the Web Speech API gave us great insights into handling asynchronous microphone streams in the browser.
What's next for VDUX: AI-Powered Military Drone Command Interface
Next steps include supporting multiple drones simultaneously (swarm control) within the same interface. We also plan to integrate real satellite mapping data (like Mapbox or Google Maps API) underneath the tactical overlay, and eventually, provide a backend bridging layer to translate these AI commands into actual MAVLink protocols for controlling real-world physical drones.
Built With
- api
- css
- google-gemini
- javascript
- web-speech-api