Inspiration
Most current AI assistants are trapped inside web browser chat windows—they can write poetry or explain concepts, but they cannot directly touch your computer, organize files, write code into workspaces, or adjust your system volume. Inspired by the vision of Tony Stark's JARVIS, we wanted to build Maya: an ultra-fast, autonomous multimodal AI desktop companion that bridges natural human voice with direct operating system execution and creative workflows in real-time.
What it does
Maya is a full-fledged agentic desktop operating companion equipped with 198 registered system tools:
- Autonomous Multi-File Coding & Scaffolding: From a single voice command, Maya can break down requirements, create directories on your desktop, write clean HTML/CSS/Python code, and launch it directly in your browser without human intervention.
- Multimodal Neural Art Generation: Transforms creative voice directives into high-definition digital assets using multi-tier neural pipelines, saving and opening them instantly.
- Real-Time OS Automation & Hardware Control: Controls system audio, Wi-Fi, Bluetooth, mobile hotspot, displays, battery diagnostics, application launching, and screen OCR analysis.
- True Multilingual Intelligence: Fluent in English, Gujarati (ગુજરાતી), and Hindi (हिन्दी) with dynamic emotion-modulated neural speech synthesis.
- Zero-Latency Local Fast-Path: Instantaneous sub-millisecond execution for deterministic system queries (time, date, battery, radio toggles).
- Encrypted Long-Term Memory Vault: Stores user facts, preferences, and chat history locally encrypted using AES-256 Fernet.
How we built it
- Core LLM & Reasoning Engine: Powered by NVIDIA's open-source Nemotron models (
nvidia/llama-3.1-nemotron-70b-instruct) accelerated through Nebius Token Factory for ultra-low latency inference and high-throughput agentic tool-calling. - Multi-Agent Orchestrator: Dynamic intent classifier and domain router that selects compact tool subsets to maintain high execution reliability.
- Speech & Audio Pipeline: Speech Recognition paired with Edge Neural TTS for lifelike emotional cadence and pitch modulation.
Challenges we ran into
- Tool-Calling Latency vs. Voice Realism: Voice conversations feel awkward if latency exceeds 2 seconds. By leveraging Nebius Token Factory's inference speed and engineering a local zero-latency fast-path for deterministic system queries, we slashed response times to sub-second conversational speeds.
- Preventing Audio Feedback Loops: Running voice input alongside TTS playback often causes the microphone to hear itself. We engineered an adaptive muting and speech-lock state machine so Maya speaks cleanly without self-interruption.
- Robust Multilingual Intent Extraction: Ensuring Gujarati phonetics and colloquial requests cleanly map to low-level Windows API commands.
Accomplishments that we're proud of
- Successfully orchestrating 198 native tools across developer tools, OS settings, smart home, and creative art generation.
- Enabling Maya to autonomously write, create, and launch a complete dashboard web app on the desktop from a single spoken sentence.
- Native, expressive Gujarati voice interaction coupled with encrypted local memory.
What we learned
- How high-throughput inference engines like Nebius Token Factory unlock seamless agentic workflows with NVIDIA Nemotron models.
- Designing resilient fallback chains for function calling and auto-recovering from malformed LLM tool arguments.
What's next for Maya
- Full touchless webcam hand gesture tracking for hands-free spatial desktop manipulation.
- Android smartphone pairing via local ADB bridge for seamless phone-to-PC notification forwarding and SMS automation.
- Proactive daily briefings and automated calendar scheduling.
