Inspiration
Most AI assistants are boxed into a chat window. They can talk about your work but can’t do it. We wanted something closer to JARVIS: an assistant you can speak to that acts on your machine, not just one that replies. That became Barq, a personal desktop AI that works as an always-on operator.
What it does
Barq takes a spoken or typed command and turns it into real action. It can run system tasks like executing processes and managing files, scrape live data from the web, and automate multi-step tasks. For anything conversational, it answers directly. Replies come back as natural, low-latency speech, and everything sits behind a 3D interactive interface so the assistant has a visual presence instead of a plain text box. It also includes wake-word activation, screen context, and vision modules so it can respond hands-free and understand what’s on your screen.
How we built it
The backend is Python and handles the full pipeline: listening, reasoning, tool execution, and speaking. Commands go to a cloud-hosted LLM, which decides whether the request is a system task, a web task, or a conversation, and routes it to the right execution path. Voice output is generated with ElevenLabs. The frontend is built with Next.js and Three.js, and communicates with the Python server in real time. All reasoning runs through cloud APIs, so the assistant stays fast even on modest hardware.
Challenges we ran into Latency: A voice assistant feels broken if it pauses too long. Keeping the loop from speech input to LLM reasoning to speech output fast enough to feel conversational took careful tuning. Safe system access: Giving an AI full control of a machine is powerful and risky. We had to think carefully about command execution, permissions, and what happens when the model gets something wrong. Connecting many parts: Wake-word detection, speech input, the LLM, tool execution, voice output, and a 3D UI all had to stay in sync across a Python backend and a JavaScript frontend. Accomplishments that we’re proud of A working end-to-end assistant: speak a command, watch it execute, and hear the response. A 3D interface that makes the assistant feel like a presence rather than a chatbot. A lightweight architecture with no local model load, so it runs smoothly on regular hardware. Real capabilities beyond conversation: system control, web scraping, and automation in one assistant. What we learned
Speed shapes the experience as much as intelligence does, and a fast, slightly simpler response beats a slow perfect one in voice interaction. We also learned that tool-using agents need guardrails from the start, because the more access you give an assistant, the more the design has to account for mistakes. Finally, we saw how much a good interface changes how people feel about an AI.
What’s next for Personal-Ai-Assistant Persistent memory across sessions, so Barq remembers your preferences and past tasks A plugin system so anyone can add custom tools and skills Stronger vision and screen understanding, including camera input A permission and sandboxing layer for safer system execution A mobile companion app for controlling your desktop remotely
