It all started with Tony Stark and his JARVIS we all looked at Iron Man and thought the same thing: Why can't our computers actually work like that? Not just a chatbot that answers questions, but an intelligent system that understands us, sees what we're doing, remembers things, and can actually take action.
Obviously, we weren't going to build JARVIS overnight. So we started experimenting. First with Ollama, then with APIs, and then what began as a simple multi-agent AI project gradually evolved into Nyth, a local-first autonomous AI system designed to turn a computer from something you operate into something that can work with you.
Nyth combines autonomous AI agents, persistent memory, vision, voice, and computer control into one system. Instead of simply telling us how to do something, Nyth can understand a goal, reason about it, interact with the computer, observe what happens, and continue working toward the result.
We built Nyth as a modular multi-agent system, with different agents responsible for different parts of the problem, including reasoning, memory, vision, monitoring, and execution.
We developed a persistent memory system so Nyth could maintain context across interactions rather than forgetting everything after a conversation. We added vision so it could understand what was happening on the screen, voice for more natural interaction, and computer-control capabilities so it could move beyond simply generating responses.
We also focused heavily on making the system local-first, keeping the computer and its environment at the center of the experience.
One of the biggest things we learned is that making an AI that can answer and respond is relatively easy compared to making one that can answer correctly and act reliably.
An autonomous system has to plan, choose the right tools, understand the results of its actions, recover from failures, and maintain context throughout the process.
We learned a lot about agent orchestration, memory systems, tool calling, computer vision, state management, and handling failures. More importantly, we learned that autonomy isn't just about giving an AI more tools. It's about teaching it when and how to use them.
The hardest part was reliability.
Real computers aren't predictable. Windows changes, applications behave differently, buttons move, actions fail, and sometimes the AI gets something wrong. We had to build systems that could recognize when something went wrong and adapt instead of completely falling apart.
We also had to balance autonomy with user control. We wanted Nyth to be capable of doing things independently without turning the user's computer into an AI playground with no supervision. Because apparently giving an AI control of your computer might just not be the best choice and we had to learn that the hard way.
Nyth is our attempt to take the idea that inspired us from fiction and turn it into something real.
We aren't claiming we built JARVIS.
We built the beginning of our own version of that idea: an AI that doesn't just talk to you, but works with you.
Log in or sign up for Devpost to join the conversation.