Inspiration
I first watched Iron Man when I was around 8 years old, and JARVIS instantly became the coolest part of it for me. It wasn't just the AI itself, it was the idea of having something that could give you daily briefs, understand what was happening, control things around you and get stuff done without you having to do every single step yourself.
Since then I always wanted my own JARVIS. So going from being a kid thinking about how cool that would be, to being 20 and actually having my own version running on my computer is probably why this project means so much to me. I didn't want to build another chatbot with a futuristic interface, I wanted something that could actually listen, remember, use tools and do things for me.
What it does
JARVIS is a voice-first personal AI assistant that can search the web, control the browser, remember context, use APIs, schedule tasks, work with files, handle voice input and output, and complete requests that need multiple steps. It can use both local and cloud AI models depending on what I need, so it is not tied to just one model or one type of task.
I also pushed it into more real-world workflows. JARVIS can help with bookings for restaurants, hotels, clinics, salons, activities and other appointments, prepare phone calls through Twilio, and stop before any sensitive or final action so I stay in control. It can also launch hands-free through two claps or the local phrases "Hey Jarvis, daddy's home" and "Hey buddy, focus here".
How we built it
JARVIS is mainly built in Python. The AI handles the reasoning side, while separate tools handle things like browser control, web search, memory, scheduling, voice, files, APIs and other actions. I kept these abilities separate so I could keep adding new features without turning the whole project into one massive script.
I added cloud models like Gemini, local wake phrase recognition using Vosk, speech recognition, text to speech, Twilio calling, booking approval rules and Windows-specific reliability features. A lot of the work was not just adding features, but making all of these different systems actually work together like one assistant.
Challenges we ran into
The biggest challenge was reliability. Getting an AI model to answer something is pretty easy compared to getting it to understand a task, choose the correct tool, use it properly, remember what happened and continue correctly. Memory, browser control and voice were some of the hardest parts, and there were bugs that took hours or even days to track down.
I also had to deal with Windows audio routing, microphone issues, API failures, local model limits, Gemini session errors, background process crashes and Twilio trial restrictions. Once JARVIS started doing real actions, permissions became important too, because knowing when not to act is just as important as knowing what to do.
Accomplishments that we're proud of
The biggest thing I am proud of is that JARVIS actually feels like more than a chatbot now. It has memory, tools, web access, browser control, scheduling, local and cloud models, voice interaction and real-world actions all working through the same assistant.
I am also proud of the smaller things that made it actually usable every day, like fast startup, reliable clap detection, local wake phrases, fixing microphone routing, restoring an existing window and stopping duplicate processes. JARVIS is mostly complete in terms of what I originally wanted to build, so now I am mainly improving reliability and adding more useful integrations.
What we learned
The biggest thing I learned is that the AI model is only one part of building an assistant like this. You also need memory, tools, APIs, browser control, scheduling, audio processing, permissions, error handling and operating system integration before it starts feeling like an actual assistant.
This project also forced me to learn a lot more Python because every new feature brought a completely different problem. Probably the biggest lesson was that making AI talk is the easy part. Making it remember, listen, use tools, control things, know when to stop and recover when something breaks is where most of the real work is.
What's next for JARVIS
JARVIS is mostly complete in terms of what I originally wanted it to be, so now the focus is on making what already exists even better. I want to improve browser reliability, bookings, calls, memory and recovery when something fails halfway through a longer task.
I also want to keep adding tools whenever I find something in my own daily life that makes me think I shouldn't still be doing this manually. The goal is still basically the same one I had when I first watched Iron Man, tell JARVIS what I need and have it take care of as much of the rest as possible.
Credits / Attribution
JARVIS originally started from work by Faith Makes. I downloaded his repository when I first started experimenting with the idea and used parts of the original code as my starting point. Some of the UI direction was also inspired by his version.
As I kept developing mine, I moved over the parts I needed and rebuilt most of the project around what I wanted JARVIS to do. The current version has been heavily changed and expanded with my own tools, memory work, booking and calling systems, hands-free activation, Windows integration and reliability improvements, but I still want to properly credit Faith Makes for the original foundation and UI inspiration.
Log in or sign up for Devpost to join the conversation.