-
-
Jarvis in its full capabilities
-
Jarvis Active
Inspiration
I first watched Iron Man when I was around 8 years old, and JARVIS instantly became the coolest part of it for me. It wasn't just the fact that it was an AI, it was the idea of having something that could give you daily news, understand what was happening around you, control your systems and get things done without you having to do every single step yourself. Since then I always wanted my own JARVIS, and when they killed him in Age of Ultron, 9 year old me was definitely not happy about it.
So in a weird way, this project became my attempt at bringing JARVIS back. Going from being a kid watching Iron Man and thinking someday I want something like that, to being 20 and actually having my own version running on my computer is probably why this project means so much to me. I didnt even want to think of building another chatbot with a futuristic UI. I wanted to build something that could actually listen, remember what i told it, use tools, control things around me and slowly become the JARVIS I imagined when I was younger.
What it does
That childhood idea basically became the goal for the whole project. JARVIS is a voice-first personal AI assistant that can search the web, control the browser, remember context, use APIs, schedule tasks, run OCR on uploaded PDF's and scanned documents, Handle voice input and output, and work through requests that need multiple steps. It can use both local and cloud AI models depending on the task, and the main idea is that I should be able to tell JARVIS what I need without having to explain every step of how to do it.
As I kept building it, I started pushing it into more real-world tasks too. JARVIS can help with bookings for restaurants, hotels, clinics, activities, events and other appointments, and it can use files I specifically allow it to access when information is needed. It stops before sensitive or final actions like confirmations, payments, deposits or using private information. I also added phone calling through Twilio, where JARVIS prepares the call, shows me the details first and waits for approval before dialing. On top of that, I wanted it to actually feel hands-free, so two claps can launch it and the local phrases "Hey Jarvis, daddy's home" and "Hey buddy, focus here" can wake or restore it.
How we built it
To make all of that work, JARVIS is mainly built in Python with separate tools handling different abilities. The AI model handles the reasoning side, while tools handle web search, browser control, memory, scheduling, voice, APIs and other actions. I kept those abilities separate on purpose because as JARVIS grew, I didn't want every new feature to mean changing one huge script and hoping nothing else broke.
A lot of the later work was about making everything around the AI actually reliable. I added local and cloud model support, local wake phrase recognition using Vosk, approval rules for real-world actions, Twilio calling, Windows startup support, microphone routing fixes, duplicate-instance protection and a way to restore the existing JARVIS window instead of opening another one. Most of these aren't the kind of things you notice in a demo, but they are what made the project start feeling like something I could actually leave running on my computer every day.
Challenges we ran into
That was also where most of the difficulty came from. Getting an LLM to answer a question is honestly the easy part. Getting it to understand what I want, choose the correct tool, use it properly, remember what already happened and continue correctly is much harder. Memory was one of the first big problems because JARVIS could have a proper conversation and then suddenly lose the context. Browser control also gave me a lot of trouble because websites are never completely consistent, and voice made every small delay much more noticeable.
Then there were all the random problems that come with connecting everything together: Windows audio issues, microphone routing, local model limits, Gemini session errors, API failures, background process crashes and Twilio trial restrictions. Some bugs took a few hours to fix and others kept me stuck for almost two days. I also had to think a lot more about permissions, because once JARVIS can actually perform actions, a mistake matters more. A wrong answer is annoying, but a wrong booking or payment is a completely different problem.
Well am proud to say after sorting out most of these issues I recently used JARVIS to book my own car rental with my selected model, to making the payment to taking responsibility so that am able to collect it all from one command.
Accomplishments that we're proud of
The biggest thing I am proud of is that JARVIS actually feels like more than a chatbot now. It has memory, web access, browser control, scheduling, tools, local and cloud models, voice interaction and real-world actions all working through the same assistant. The booking and calling systems are probably some of my favorite additions because they show the difference between AI just telling me something and AI actually helping complete the task.
But some of the smaller things probably made me just as happy. Getting it to start quickly, clap detection working properly, local wake phrases, fixing the microphone, bringing an existing window back into focus all made it feel more complete. JARVIS is now mostly at the stage I imagined when I first started building it, and being able to clap twice and see something I wanted when I was 8 years old open on my screen still feels kinda unreal.
What we learned
Building JARVIS completely changed how I look at AI assistants. At the start I thought the hardest part would be making the model smarter, but I learned pretty quickly that the model is only one piece of the whole system. You need memory, tools, APIs, scheduling, browser control, audio processing, permissions, error handling, local networking and operating system integration before it starts feeling like a real assistant.
The project also forced me to learn a lot more Python because every new feature threw me into a different problem. One day I was working on agent logic and the next I was debugging Windows audio devices or webhooks. Probably the biggest lesson was that making AI talk is the easy part. Making it remember you, listen properly, use tools, control things, know when to stop and recover when something breaks is where most of the real work is.
What's next for JARVIS
JARVIS is mostly complete in terms of what I originally wanted it to be, so now I don't want to completely change the project. I mainly want to keep improving what already works, especially browser actions, bookings, calls, memory and recovery when something fails halfway through a longer task.
I will also keep adding tools whenever I find something in my own life that makes me think I shouldn't still be doing this manually. That is pretty much how JARVIS has grown from the start. The goal is still the same one I had when I first watched Iron Man: I want to be able to talk to JARVIS, tell it what I need and have it take care of as much of the rest as possible. The flying suit can come later.
Credits / Attribution
JARVIS originally started from work by Faith Makes. I downloaded his repository when I first started experimenting with the idea and used parts of the original code as my starting point. Some of the UI direction was also inspired by his version.
As I kept developing mine, I moved over the parts I needed and rebuilt most of the project around what I wanted JARVIS to do. The current version has been heavily changed and expanded with my own tools, memory work, booking and calling systems, hands-free activation, Windows integration and reliability improvements, but I still want to properly credit Faith Makes for the original foundation and UI inspiration.
Log in or sign up for Devpost to join the conversation.