JARVIS

Inspiration

Ever since I was a kid, my father and I would sit down to watch a movie together. It wasn’t until I was 8 that I watched Iron Man for the first time, and I was instantly captivated by Tony Stark’s AI assistant, J.A.R.V.I.S. The idea of having a helpful, intelligent companion inspired me to pursue engineering. Now, at 21, while exploring ideas for a personal project, I noticed how AI is rapidly becoming a part of everyday life. I thought back to the first AI I had ever admired J.A.R.V.I.S. and decided to build my own personal AI assistant for the Macbook.

What it does

JARVIS is your personal Macbook assistant, designed to simplify everyday tasks and streamline productivity. Starting with the basics, he can tell you the time, date, and weather at your location, search the web, open links, play Apple Music, and set timers. Beyond that, JARVIS handles communication: he can send emails, respond to text messages, and assist with phone calls.

With memory integration, JARVIS can create events and reminders, announcing them when needed. He can provide ETA for destinations, and if you have smart plugs, he can turn them on or off. He also includes restart, power down, mute, and unmute functionality.

For developers, JARVIS takes productivity to another level. He can create or open projects in Visual Studio Code, generate new files or folders, write code within specified files, assist with documentation, and fully manage source control. He can initialize Git repositories, stage and unstage changes, commit, and push to remote repositories. JARVIS learns context over time, providing increasingly personalized assistance. Think of him as a seamless combination of ChatGPT and GitHub Copilot, right on your Macbook.

How was it built

JARVIS was built as a personal Macbook assistant using Python and AppleScript. For real-time interactions, I leveraged Livekit and Mem0. For the AI model, I chose OpenAI, and for text-to-speech (TTS), I used Cartesia to closely mimic the voice of J.A.R.V.I.S. from Iron Man. Web searches are powered by SERPAPI, emails by SMTP, and voice output through Cartesia. This combination allowed me to build a fully functional AI assistant without relying on costly paid services.

Challenges we ran into

One major challenge was funding. As a student at SMC without a steady income, I had to find cost-effective ways to build and run JARVIS without overspending. This constraint influenced my decision to implement much of the functionality through AppleScript and rely on free tiers for OpenAI and Cartesia. Balancing functionality with affordability was a constant challenge, but it forced me to be creative and efficient in the architecture and services I chose.

Accomplishments that we're proud of

Hearing JARVIS speak and being able to rely on him for coding assistance is a huge personal achievement. Seeing him handle tasks like creating projects, generating code, and managing Git repositories gave me a real sense of accomplishment. Beyond that, I’m proud of having built a functioning AI assistant entirely on my own, bridging multiple technologies seamlessly.

What we learned

Through this project, I learned how to integrate real-time AI, text-to-speech, and automation tools to create a practical assistant. I gained hands-on experience with Livekit, OpenAI Realtime, AppleScript, and Python automation. Most importantly, I learned how to take an idea inspired by a movie and turn it into a tangible, functional AI assistant.

What's next for JARVIS

Moving forward, I hope to expand JARVIS’s capabilities by integrating more APIs and libraries, enabling him to interact with a broader ecosystem of software and services. I also aim to implement security features, ensuring that all data handled by JARVIS is safe and private. Eventually, I envision a fully adaptive, context-aware assistant capable of anticipating needs and proactively assisting with both daily tasks and complex workflows.

Built With

• api
• applescript
• cartesia
• google
• openai
• openweathermap
• python